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Summary 


STEM Pathways is a partnership between five informal STEM organizations—The 
Bakken Museum, Bell Museum of Natural History, Minnesota Zoo, STARBASE 
Minnesota, and The Works—with the Minneapolis Public Schools (MPS) and the 
Minnesota Department of Education (MDE). The purpose of this collaborative effort is to 
raise the access, enthusiasm, and academic achievement of young people in STEM and 
their preparation for future STEM careers through a deliberate and interconnected system 
of STEM learning opportunities. Further, STEM Pathways tests a model for collaboration 
and its potential for expansion and replication across more grade levels, schools, 
organizations and communities. 


As part of a three-year pilot project, STEM Pathways partner organizations (informal 
STEM education organizations working with MPS and MDE) began the first year in 
2013-14 with an intensive collaboration process to learn about each other’s programs, 
make concept and theme connections, develop shared tools and strategies, and plan for 
project implementation. STEM Pathways began offering activities to fourth- and fifth- 
grade students at six MPS elementary schools in 2014-15 and continued in 2015-16. 


Evaluation 


A core component of the STEM Pathways project is ongoing evaluation to inform 
programming as well as future efforts for the STEM Pathways collaborative. STEM 
Pathways participates in rigorous, independent evaluation conducted by Wilder Research 
during the three-year pilot project. Evaluators assess the implementation and outcomes of 
the project using a student survey, small-group interviews, academic data, a teacher survey, 
and interviews with MPS leaders and STEM Pathways partner representatives. This report 
summarizes outcome findings from student surveys and academic data over the two years 
of program implementation (2014-15 and 2015-16), as well as evaluation findings from 
data collected through surveys and interviews with students, MPS teachers and leaders, 
and partners during the second program implementation year. 


Student survey results 


A survey was administered to fourth- and fifth-grade students in the six STEM Pathways 
schools in MPS in fall and spring. The survey assessed students’ STEM awareness, 
attitudes, interests, and activities. Results are reported for 255 Cohort 1 students who 
completed the survey in fall 2014, spring 2015, and spring 2016, and for 345 Cohort 2 
students who completed the survey in fall 2015 and spring 2016. Changes in students’ 
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responses in these areas over time may be associated with participation in STEM Pathways. 
In 2015-16, Cohort 1 students were fifth graders and Cohort 2 students were fourth 
graders. Cohort 2 survey responses are compared to those of Cohort | in fourth grade. 


Results by topic area 


Survey results are summarized by the following topic areas: awareness and relevance of 
STEM, STEM interest and confidence in STEM abilities, interest in STEM subjects, application 
of STEM to problem solving, and careers using STEM. The survey items were a series of 


99 66 99 66 


statements with the response options: “agree a lot,” “mostly agree,” “agree a little,” 
“don’t agree,” and “don’t know.” Those who agreed a lot or agreed mostly were considered 


to be in agreement with an item. 


Overall, the comparison between fourth-grade survey results of Cohort 2 in 2015-16 and 
Cohort 1 in 2014-15 indicates that Cohort 2 had stronger results. For Cohort 1, survey results 
were stronger overall at the end of their fifth-grade year (spring 2016) than at the end of 
their fourth-grade year. Figures 1-8 illustrate some of the findings. Results in detail can 
be found in STEM Pathways Student Survey Results through 2015-16 School Year 
(Mueller & Gozali-Lee, 2016). 


Awareness and relevance of STEM 


Students’ awareness of STEM and belief in its relevance increased in both cohorts. For 
Cohort 2 fourth graders, agreement increased significantly from fall to spring for three of 
the four items in this topic area. These three items concerned the importance of STEM 
knowledge to students’ futures, their awareness of STEM in the world around them, and 
their knowledge of STEM-related out-of-school activities. In Cohort 1, agreement with 
the first two items also increased significantly in the fourth-grade year and continued to 
increase slightly in the fifth-grade year. Agreement with the third item (their knowledge 
of STEM-related out-of-school activities) increased significantly in fifth grade and was a 
significant increase over the two-year assessment period. Agreement with a fourth item in 
this topic area (frequently doing STEM-related out-of-school activities) did not change 
significantly in either cohort over time. 
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1. Cohorts 2 and 1 in fourth grade: STEM knowledge is very important to 
my future 
Percentage of students who “agree a lot” or “mostly agree” 


76% 75% 
64% nae 


= Fall 
= Spring 


2015-16* 2014-15* 
Cohort 2 Cohort 1 
(N=334) (N=343) 


*The change in agreement with the item from fall to spring is statistically significant (p<.05). 


2. Cohort 1 in fourth and fifth grade: STEM knowledge is very important to 
my future ? 


Percentage of students who “agree a lot” or “mostly agree” 


1% 2" 


66% 


Fall Spring Spring 
2014 2015 2016 


Note: The change in agreement with the item from fall 2014 to spring 2015, and from fall 2014 to spring 2016, are statistically 
significant (p<.05). 


@ Includes 246 students who responded to this item at all three time points. 


STEM interest and confidence in STEM abilities 


Agreement with liking to learn STEM and being really good at STEM increased 
significantly from fall to spring among Cohort 2 fourth graders. In Cohort 1, agreement 
with liking to learn STEM increased significantly in fourth grade but agreement with 
being really good at STEM decreased significantly. However, agreement with this latter 
item increased significantly in fifth grade so that the level of agreement at the end of fifth 
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was nearly the same as at the beginning of fourth grade. Agreement with wanting to do 
more STEM-related activities and thinking one would be good at a job that uses STEM 
did not change significantly in either cohort over time. 


3. Cohorts 2 and 1 in fourth grade: | am really good at STEM 


Percentage of students who “agree a lot” or “mostly agree” 


See 
0 


48% 
= Fall 
= Spring 


2015-16* 2014-15* 
Cohort 2 Cohort 1 
(N=338) (N=343) 


*The change in agreement with the item from fall to spring is statistically significant (p<.06). 


4. Cohort 1 in fourth and fifth grade: | am really good at STEM ? 


Percentage of students who “agree a lot” or “mostly agree” 


59% 55% 


eh 


Fall Spring Spring 
2014 2015 2016 


Note: The change in agreement with the item from fall 2014 to spring 2015, and from spring 2015 to spring 2016, are 
statistically significant (p<.05). 


a Includes 249 students who responded to this item at all three time points. 
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Interest in STEM subjects 


Students were asked if they liked learning each of the four STEM subjects. The percentage 
of fourth graders in Cohort 2 and Cohort 1 who liked learning engineering increased 
sharply from fall to spring. In Cohort 1, the increase in agreement seen at the end of 
fourth grade was maintained at the end of fifth grade. Relatively high percentages of 
fourth graders in both cohorts liked learning the other three STEM subjects (science, 
technology, and math) in both the fall and the spring. In Cohort 2, agreement with liking 
science decreased a small (but statistically significant) amount from fall to spring. In 
Cohort 1, the percentage who liked learning technology increased significantly from the 
beginning of fourth grade to the end of fifth grade. 


Application of STEM to problem solving 


Agreement with applying STEM to problem solving increased significantly from fall to 
spring among Cohort 2 fourth graders, which concerned using technology to solve problems 
and thinking like an engineer to design solutions to problems. Among fourth graders in 
Cohort 1, there was a significant increase in agreement with the engineering item but not 
with the technology item. The increase in the engineering item was maintained through 
fifth grade. 


5. Cohorts 2 and 1 in fourth grade: | think like an engineer to design 
solutions to problems 


Percentage of students who “agree a lot” or “mostly agree” 


One 58% 
50% 


= Fall 
= Spring 


2015-16* 2014-15* 
Cohort 2 Cohort 1 
(N=334) (N=339) 


*The change in agreement with the item from fall to spring is statistically significant (p<.06). 
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6. Cohort 1 in fourth and fifth grade: | think like an engineer to design 
solutions to problems # 


Percentage of students who “agree a lot” or “mostly agree” 


57% 59% 


— 


Fall Spring Spring 
2014 2015 2016 


Note: The change in agreement with the item from fall 2014 to spring 2016 is statistically significant (p<.05). 


a Includes 246 students who responded to this item at all three time points. 


Careers using STEM 


In both cohorts, fourth graders’ agreement increased significantly for the item concerning 
knowing about many jobs that use STEM (Figure 7). In Cohort 1, this agreement continued 
to increase in fifth grade (Figure 8). There was no significant change for fourth graders in 
either cohort for the two other career-related items: thinking one would be good at a job 
that uses STEM (mentioned above) and wanting to have a job that uses STEM. The level 
of agreement with these items did not change over the two school years in Cohort 1. 


7. Cohorts 2 and 1 in fourth grade: | know about many jobs that use STEM 


Percentage of students who “agree a lot” or “mostly agree” 


68% 67% 
57% 


= Fall 
= Spring 


2015-16* 2014-15* 
Cohort 2 Cohort 1 
(N=325) (N=334) 


*The change in agreement with the item from fall to spring is statistically significant (p<.05). 
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8. Cohort 1 in fourth and fifth grade: | know about many jobs that use STEM 2 


Percentage of students who “agree a lot” or “mostly agree” 


69% 74% 
{0} 


61% 


Fall Spring Spring 
2014 2015 2016 


Note: The change in agreement with the item from fall 2014 to spring 2016 is statistically significant (p<.05). 


a Includes 237 students who responded to this item at all three time points. 


Overall results of the student survey do not indicate advantages for underrepresented 
students. 


Small-group interviews with students 


Small-group semi-structured interviews were conducted with fifth-grade students who 
participated in STEM Pathways in both fourth and fifth grade. Forty-four students were 
interviewed in an effort to gain a more in-depth understanding of their experiences in 
STEM Pathways and attitudes towards STEM. 


Results from small-group interviews indicate that: 


m STEM Pathways experiences are fun, different from and better than other field trips 
and classroom visits. 


m= Students made connections between their STEM Pathways experiences and their 
classroom learning. 


m= Students are more confident in their STEM knowledge because of participating in 
STEM Pathways. 


= Students are more interested in STEM careers because of participating in STEM 
Pathways. 


= Students have positive feelings about STEM, in general. 
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While the overall results from the student interviews are positive, the results also pointed 
to areas for program improvement. These include: 


m= Students have varied understandings of STEM Pathways. 
= Students did not recognize that their STEM Pathways experiences involved math. 


= Students recognize primarily surface-level connections between the programs 
provided by STEM Pathways partners. 


= Students recognized they still have a lot to learn about STEM, and many see STEM 
learning as challenging, which creates mixed feelings related to their confidence in 
their abilities to do STEM. 


Student achievement results 


STEM Pathways’ long-term goal is to increase student academic achievement in STEM, 
especially in underrepresented groups (low-income students, racial/ethnic minorities, and 
females). Evaluators compared STEM Pathways students with similar peers attending 
other MPS schools on Minnesota Comprehensive Assessments, Series II] (MCA-IIT) 
math, reading, and science scores, and on student attendance. 


Overall, there was no evidence found at this point for an academic achievement advantage 
among students who participated in STEM Pathways. Results indicated that differences 
between the treatment and comparison groups in spring 2016 MCA-III performance and 
one-year MCA-III growth tended to be small and not statistically significant in almost all 
instances. Although the differences were small, they generally favored the comparison 
group over the treatment group. This pattern occurred in all three cohorts for the subjects 
assessed (math, reading, and science). Some of the differences in academic achievement in 
favor of the comparison group over the treatment group were stronger and statistically 
significant for some demographic groups. School attendance rates in 2015-16 were the 
same or almost the same in the treatment and comparison groups in all three cohorts. 


Results of Minneapolis Public Schools leader interviews 


Wilder Research interviewed MPS leaders to get their views on the implementation and 
accomplishments of STEM Pathways, including its benefits to the district, teachers, and students, 
as well as suggestions for changes or improvements. Eight MPS leaders participated, including 
five principals of the six STEM Pathways schools. Leaders viewed their collaboration with 
STEM Pathways positively. 
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Several benefits to teachers, students, and the schools were mentioned, including the 
following: 


= Increasing student enthusiasm for STEM and providing exposure to STEM careers. 
= Increasing student enthusiasm for science and math classes in school. 


= Providing learning opportunities to students who might not have access to the STEM 
activities outside of school. 


= Providing materials and communication to teachers to highlight areas of connection 
between STEM Pathways activities, state standards, and the MPS curriculum. 


= Providing teaching materials and supports, as well as inspiring teachers to build on 
the motivation and excitement in their own teaching of STEM. 


= Helping teachers integrate STEM into other areas of the curriculum, such as literacy. 


= Increasing teacher awareness of opportunities for professional development that could 
improve the way they teach or integrate STEM in classrooms. 


m= Enhancing schools’ ability in delivering STEM education. 


All leaders said they would like STEM Pathways to continue, and most would like STEM 
Pathways to expand to other grades and schools. 


Results of teacher survey 


Wilder Research conducted a web survey with fourth- and fifth-grade teachers from the 
six STEM Pathways schools in spring 2016. The survey asked teachers about their 
experiences with STEM Pathways and its impacts on them, their students, and their 
school. Twenty teachers (59%) completed the survey. 


All teachers in the survey indicated that STEM Pathways: 
= Supports the state standards and the MPS curriculum in meaningful ways. 
m= Had a positive impact on their school, on them as a teacher, and on their students. 


= Increased students’ interest in STEM learning and improved their learning in STEM 
subjects. 


All teachers indicated that they would like their school to participate in STEM Pathways 
again next year. 
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Results of STEM Pathways partner representative interviews 


Wilder Research also interviewed twelve representatives from the STEM Pathways 
partner organizations, including steering committee and implementation team members. 
Representatives were asked about their views on the STEM Pathways implementation 
and accomplishments, including impacts on their organization and implications for 
informal STEM education organizations. 


Representatives of the STEM Pathways partner organizations indicated several successes 
from the second year of implementation including: 


= Better articulation of the primary goal (compared to the previous year’s evaluation 
findings), which is to increase student engagement and interest in long-term STEM 
learning and increase student interest in STEM careers. 


= Partners’ enthusiasm and commitment to working together. 


= Alignment of STEM Pathways lessons with the state math and science standards and 
MPS curriculum. 


= Development of common language and shared messages across organizations. 
Being a part of the collaborative gives partners the following benefits: 


= Networking opportunities with other informal education organizations, discussing 
common goals, sharing successes and challenges and learning from each other, and 
brainstorming new ideas. 


= Better visibility and credibility. 
= Opportunity to be part of research and evaluation work. 


= Opportunity to work more closely with school partners. 


Issues to consider 


Overall, STEM Pathways was successfully implemented in the second year. Student 
survey results were stronger for both cohorts, and students in the small-group interviews 
reported that they were more interested and more confident in STEM because of their 
participation in STEM Pathways. Results also suggest that more attention may be needed 
to advance STEM interest, knowledge, and attitudes of underrepresented students, in part 
because their baseline interest and confidence tend to be lower than that of students not 
from underrepresented groups. It also seems important to communicate clearly to students 
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and teachers about STEM Pathways and its goals, including the interconnected concepts 
among STEM Pathways partner programs and the use of math in STEM Pathways 
experiences. Findings also highlight the complex and challenging nature of STEM 
learning, including that students may need encouragement to develop confidence in their 
STEM abilities, such as recognizing their efforts, progress, and growing skills. It seems 
important to continue providing programming to advance STEM interest, knowledge, and 
attitudes of students, especially the underrepresented students. 


STEM Pathways’ potential effects on student academic achievement were examined 
using MCA tests, which are considered long-term indicators of the program impact. 
Significant achievement-related effects may be unlikely in the short term, but could 
emerge after multiple years of program exposure. 


MPS leaders and teachers viewed the project and the collaboration with the partners 
positively. Teachers wished to have more opportunities for professional development, 
including how to conceptualize STEM as an integrated discipline, and how to incorporate 
the STEM Pathways resources into classroom learning more effectively. They also 
wanted additional materials and resources, such as math and life science pages in the 
Student Portfolio, materials in Spanish, incentives for students who made learning 
progress, and more STEM professionals come to speak to classes about their careers. 


MPS leaders indicated that more coordination between teachers and partner organizations 
is important, as teachers could incorporate more about the field trip lessons in their 
classrooms. Leaders also recommended more communication with teachers ahead of time 
regarding lessons and expectations. Recognizing the many benefits to students and teachers, 
MPS leaders wished that more schools and grade levels could receive STEM Pathways 
programming. 


Partners had several ideas for strengthening STEM Pathways, including more opportunities 
to grow professionally. Partners also aspire to work collaboratively to develop best practices 
for STEM learning in informal settings, share outcomes and measurements, and to 
become an expert in the STEM field. 


Looking forward 


Having completed the three-year pilot project, partners continue to focus on strategic 
planning and action for the next phase. Through this process, STEM Pathways partners 
recognize the importance of and are committed to fostering a collaborative systems 
approach to inspire the next generation of STEM-literate decision-makers and problem 
solvers. The goal of STEM Pathways is to ensure that all young people have opportunities 
to engage in high-quality STEM learning experiences—over time and in a variety of 


STEM Pathways Evaluation 11 Wilder Research, December 2016 


settings—that lead them to develop the STEM mindset necessary to become the creative, 
STEM-skilled thinkers and innovators of the future. Core activities of the next stage of 
STEM Pathways include: 


= Supporting and expanding the network of informal STEM educators that STEM 
Pathways has created. 


= Facilitating access to a system of high-quality and interconnected STEM learning 
experiences for youth through informal STEM education organizations across school 
years, in collaboration with schools and districts and in alignment with and support of 
core curriculum and standards. 


= Promoting cross-organizational leadership to create and prioritize a culture of 
collaboration that builds authentic connections between organizations, people and 
programs; that articulates shared vision and goals; and utilizes shared measurement to 
evaluate progress. 


= Building effective funding and sustainability strategies. 
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Introduction 


STEM Pathways is a partnership between five informal STEM organizations—The Bakken 
Museum, Bell Museum of Natural History, Minnesota Zoo, STARBASE Minnesota, and 
The Works—with the Minneapolis Public Schools (MPS) and the Minnesota Department of 
Education (MDE). The purpose of this collaborative effort is to raise the access, enthusiasm, 
and academic achievement of young people in STEM and their preparation for future STEM 
careers through a deliberate and interconnected system of STEM learning opportunities. 
Further, STEM Pathways tests a model for collaboration and its potential for expansion and 
replication across more grade levels, schools, organizations and communities. 


The partnership was launched in 2012 with the following objectives: 


= To provide deliberate, coherent, and connected pathways of meaningful in-school and 
out-of-school STEM learning experiences that contribute to a local STEM learning 
ecosystem and lead to the achievement of shared youth outcomes. 


= To test anew model and culture for how informal STEM education organizations 
work together, emphasizing collaboration, shared vision, goals, knowledge, strategies, 
and measurements. 


STEM Pathways partners chose to test this model in fourth and fifth grades, based on 
current program offerings for schools provided by partner organizations. To identify partner 
schools, STEM Pathways worked with MPS leaders to identify and recruit schools with diverse 
student populations, where school leadership and teachers had previously demonstrated interest 
in and support for STEM education, and where at least some STEM-related community 
partnerships were in place. School principals received detailed information about STEM 
Pathways including the benefits and commitment required by their school. All six schools 
identified for participation chose to be part of this pilot project. 


During the planning year of the project (2013-14), STEM Pathways partner organizations 
worked collaboratively to: learn about each other’s organizations and programs; explore 
important topics in formal and informal STEM education fields; determine the overarching 
themes and concepts connecting the STEM Pathways partner programs; and create common 
tools, strategies, and messages that would help students experience long-term and 
interconnected STEM learning. This process is illustrated in Figure 9. The collaborative 
efforts of developing common goals, strategies, tools and measurements shared by multiple 
organizations to build interconnected learning experiences for students is a major shift in 
the culture of informal STEM education organizations, which have traditionally worked 
independently in their service to schools. 
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9. STEM Pathways collaboration process 


Building knowledge about: 


leads to dicovery of... 
Content themes, instructional practices, 


learning goals of partner programs via 
observation and lesson comparisons 
School district learning targets associated 
with state standards 

Next Generation Science Standards science 
and engineering practices 

Learning ecosystems 


Shared values and 


connecting themes: 


Engineering Design Processes (EDPs) (4th 
grade) 

Systems and impact of change (5th grade) 
STEM careers (both grades) 

Everyone can do STEM (both grades) 


made visible through... 


Instructional tools and strategies: 


Shared vocabulary for key STEM practices 
Linkages between different EDPs 

Visual tools for learning about systems and 
impact of change 

Interactive career exploration tool, 
discussions of STEM careers and broad value 
of STEM skills 

Instructional strategies that include and 
encourage all students and build confidence 


Source: STEM Pathways 
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In the second year (2014-15), STEM Pathways began offering programming in fourth- 
and fifth-grade classrooms at six schools in MPS and continued to do so in the third year 
(2015-16). The six schools participating in STEM Pathways include four K-5 schools, 
one K-8 school, and one 3-8 school. Revision of existing and creation of new instructional 
tools also occurred during the second and third years, informed by program evaluation. 


Wilder Research serves as the independent evaluator of STEM Pathways for this pilot 
project. As such, Wilder Research worked closely with the STEM Pathways partners to 
design the evaluation and with MPS to implement the evaluation activities. 


Content of the report 


This evaluation report focuses on the third and final year of the STEM Pathways pilot 
project (the second program implementation year) and includes short-term project 
outcomes for schools, teachers, and partners during the 2015-16 school year. Student 
outcomes are reported for both implementation years: 2014-15 and 2015-16. 


The report is organized into the following sections: 
= Partnership efforts 

= Research methods 

= Implementation 

= Outcomes 


= Looking forward 
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Partnership efforts 


The STEM Pathways steering committee, which consists of key representatives from 
informal STEM organizations, MDE, and MPS, has met regularly since the project began 
in 2013-14. Additionally, working meetings are held for STEM Pathways educators and 
steering committee members to learn about each other’s organizations and make connections 
between them, explore best practices in STEM education, and plan for instructional tools to 
be developed. Implementation team and grade-level workgroups meet to develop teaching 
materials, and plan and coordinate the activities for fourth- and fifth-grade students. 


STEM organizations’ leaders and educators and MPS leaders and teachers participated 

in four working meetings during the planning year. These meetings covered topics on: 
identifying the common concepts and language for Engineering Design Processes (EDPs) 
used by partner organizations; learning about the Next Generation Science Standards 
(NGSS), Minnesota science standards and MPS science curriculum learning targets; 
research-based practices for engaging students in writing about science; and developing 
the STEM Pathways logo. 


The STEM Pathways steering committee and educators continued to meet to refine and 
advance project efforts during the first program implementation year. Working meeting 
topics and task force efforts during this year included: developing the STEM Pathways 
Portfolio and Career Interactive; continuing to learn about MPS learning targets; adapting 
the concepts of natural ecosystems to imagine a local STEM learning ecosystem, including 
its components and resources to thrive; sharing best practices among partners; and 
defining STEM Pathways values and key messages. A STEM Pathways representative 
and the MPS K-8 STEM Curriculum Specialist also met with fourth- and fifth-grade 
teachers and principals at each school to provide information about the project, answer 
questions and share instructional tools. At the end of the school year, an informal focus 
group was also held with teachers to gain their perspective on project successes and 
opportunities for improvement. 


At the beginning of 2015-16, the second implementation year, evaluators shared findings 
from the first implementation year with the STEM educators and leaders and MPS leaders. 
Partners used the findings to improve programming, including revision of existing and 
development of new instructional tools as well as a teacher guide. For example, following 
the meetings, a grade-level workgroup met to strengthen activities (especially on building 
students’ confidence in STEM), share ideas for improving connections between partners, 
and discuss a career focus. Individual partner organizations further considered evaluation 
findings to varying degrees. The STEM Pathways project director met with the classroom 
teachers at each school to share evaluation findings, listen to teachers’ interpretations of 
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findings, and encourage use of STEM Pathways instructional tools. Responsive interviews 
were conducted with some students by an evaluation consultant in the middle of the 
program year to gauge their attitudes toward and understanding of their experiences 
learning STEM in general and with STEM Pathways specifically. The interviews were 
used to inform programs and program delivery. 


The steering committee met ten times throughout the second program year to discuss the 
future of STEM Pathways beyond the pilot year and participate in strategic planning. The 
steering committee also worked with evaluators to add small-group interviews with students 
to the formal evaluation plan, based on the findings from first-year student survey and the 
mid-year responsive interviews, to better understand students’ perspectives on their 
STEM Pathways experiences and their attitudes toward STEM. 


STEM Pathways partner representatives have presented the work of STEM Pathways at 
several national and local professional conferences throughout the years and published in 
a refereed journal. 


Focus of work in the second program implementation year 


Partners were asked during interviews to describe STEM Pathways’ goals, and if and 
how the goals have evolved since the beginning of the project. In describing STEM 
Pathways, several partners mentioned that the primary goal is to increase student 
engagement and interest in long-term STEM learning and increase their interest in 
STEM careers. 


STEM Pathways is a collaboration of five Twin Cities informal STEM education 
institutions, Minneapolis Public Schools, and the Minnesota Department of Education to 
increase the long-term interest and academic achievement of students in STEM and to 
create an illuminated pathway to future STEM careers. — Partner interviews 


We are hoping students will become more excited about STEM as they understand all the 
ways it can be applied in their lives. They gain excitement, interest, skills, and realize 
[that] it is something they can do going forward. They see themselves in it, so they are 
developing a STEM identity. — Partner interviews 


It is intended to test and implement a sequence of STEM learning experiences provided 
by community organizations in classrooms, connecting with the formal school day, looking 
at impacts of connected experiences on both outcomes related to interest and 
engagement and career as well as academic indicators for students. — Partner interviews 


We offer either field trips onsite or offsite or both, opportunities at schools with science 
and engineering, and more insight into careers and what people actually do, with the hope 
of inspiring students toward careers in STEM. — Partner interviews 
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Partners felt that STEM Pathways has stayed true to the original primary goal, but the 
focus of work to achieve this goal has evolved since the beginning of the project. Partners 
said that the work has changed to be less about building a deliberate sequence of 
experiences, and more about providing a combination of high-quality STEM experiences 
for students. There has been more attention given to developing a model for collaboration 
between formal and informal education institutions and among informal STEM education 
organizations. Partners also recognized that, even though the focus of the pilot project has 
been on programs for schools, the collaboration has provided value for the educational 
services of partner organizations. Over the two implementation years, the purpose and 
goals of STEM Pathways have become further defined and articulated. 


| think that the goals are still similar to what they were at the origin...— Partner interviews 


| think the goals have pretty much stayed the same. We have shifted course on how we 
got to those goals, so we are focusing more on the objectives [now]. — Partner interviews 


We have always been focused on the student experience, but one of the things that 
started to become clearer is this model between the partnerships between the different 
organizations, including the school district and the Department of Education, just as a 
model for collaboration in ways that an informal education can support formal education. 
Also, really in a way that we as an informal educational organization can support each 
other and look at the way we complement each other and can work together instead of in 
competition. — Partner interviews 


| think that we broadened our scope to think about what the impact/value is of all the 
STEM education services that these organizations provide, not just the ones that occur 
during the school days. — Partner interviews 


! guess | would say that initially the idea of a deliberate or clearly sequenced set of 
experiences with instructional tools was a much stronger component, over time that has 
loosened up to not be so dependent on a set sequence of events, which was originally 
part of the model. — Partner interviews 


The goals have not so much evolved, but have become more clear and succinct. 
— Partner interviews 
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Research methods 


In consultation with the STEM Pathways steering committee, Wilder Research developed 
an evaluation plan for the two project implementation years. Research methods are 
described below. 


Research design 


Wilder Research’s evaluation has implementation and outcome components addressing 
the following goals: 


= Implementation evaluation. Assess project implementation and provide feedback 
useful for strengthening implementation efforts. 


= Outcome evaluation. Assess short-term outcomes for project activities and provide 
initial reporting on long-term indicators reflective of major project goals. 


Research questions 


The evaluation efforts were guided by the following research questions: 


Implementation questions 


The implementation component of the evaluation is an important mechanism for 
understanding project efforts that seem effective and for providing feedback useful for 
strengthening project activities. Key research questions associated with the implementation 
evaluation include the following: 


1. How successfully is the STEM Pathways model being implemented? 


2. What partner and school characteristics are associated with strong implementation? In 
what ways can implementation be strengthened? 


3. How well does the collaboration function, and how can it be strengthened? 
4. How effective is professional development, and what are its future needs? 


5. What are the core components of the program model and conditions for replication? 
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Outcome questions 


The outcome component of the evaluation addresses changes in systems, perceptions, and 
student outcomes related to STEM. During the first and second evaluation year, outcome 
measurement focused on initial system changes, short-term outcomes associated with project 
activities, and initial measurement of student outcomes reflective of long-term project impact. 
Research questions for this component of the evaluation include the following: 


6. Does the STEM Pathways model enhance the short- and long-term outcomes of 
populations underrepresented in STEM? 


7. How well does the model work for specific underserved populations, and in what 
context? 


8. What impacts does the model have on informal STEM education organizations? 
9. What impacts does the model have on classroom teachers? 


10. What are the implications for the field of informal science education? 


Data collection procedures 


Data collection procedures for the evaluation are described below. Specific research 
questions addressed through each method are denoted. 


STEM Pathways partner interviews 
(Questions 1-5, 8, 10) 


In spring 2016, Wilder Research conducted one-to-one phone interviews with STEM 
Pathways partners regarding the goals, implementation, and accomplishments of STEM 
Pathways and about suggestions for changes or improvements for STEM Pathways in the 
future. Interviews were conducted with members of the steering committee and members of 
the implementation team. Respondents included the project director for STEM Pathways as 
well as three representatives from STARBASE, and two representatives each from The 
Bakken Museum, Minnesota Zoo, The Works Museum, and The Bell Museum of Natural 
History. Similar interviews were conducted in spring 2015. 


Minneapolis Public Schools leader interviews 
(Questions 1-4, 6, 7, 9, and 10) 


Wilder Research conducted one-to-one phone interviews with MPS leaders in spring 
2016, including five of the six school principals, the Director of Elementary Education, 
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the K-8 STEM Curriculum Specialist, and K-8 STEM Curriculum Integration Specialist. 
The survey asked MPS leaders about the implementation and accomplishments of STEM 
Pathways, including its benefits to the district, teachers, and students, as well as suggestions 
for changes or improvements for STEM Pathways in the future. Similar interviews were 
conducted in spring 2015. 


Teacher survey 
(Questions 1-4, 6, 7, 9, and 10) 


Wilder Research conducted a web survey with fourth- and fifth-grade teachers from the 
six STEM Pathways schools in spring 2016. The survey asked teachers about their 
experiences with STEM Pathways, and its impacts on them, their students, and their 
school. The survey was completed by 20 out of 34 teachers (59%). Between one and five 
teachers in each school participated in the survey. Sixteen of the respondents (70%) were 
fourth-grade teachers and four (30%) were fifth-grade teachers. The teacher survey was 
not conducted in spring 2015. As mentioned earlier, STEM Pathways representatives 
held an informal focus group with teachers to gain their perspective on project successes 
and opportunities for improvement after the first year of project implementation. 


Student pre-post survey 
(Questions I, 6, and 7) 


A survey was administered to fourth- and fifth-grade students at the six STEM Pathways 
schools in MPS. The first cohort of students (Cohort 1) attended STEM Pathways schools 
for two years, in 2014-15 as fourth graders and in 2015-16 as fifth graders, experiencing 
two years of STEM Pathways programming. The student survey was administered to this 
cohort in fall 2014 (baseline assessment), and repeated in spring 2015 and spring 2016, 
providing the opportunity to examine potential changes in their survey responses over 
two school years. The second cohort of students (Cohort 2), who were fourth graders in 
the 2015-16 school year, experienced one year of STEM Pathways programming. They 
took the survey in fall 2015 (baseline assessment) and then again in spring 2016, providing 
the opportunity to examine potential changes in their survey responses from the beginning 
to the end of the 2015-16 school year. 


The survey was administered to students as a group in their classrooms by Wilder Research 
staff. After a brief explanation of the survey, Wilder Research staff read the questions and 
students provided their answers on paper-and-pencil survey forms. A few students who 
were absent on the day the survey was administered completed it later. Students’ parents or 
guardians were informed about the survey by letter and could have their child excluded 
from the survey if they wished by contacting the school. 
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Student survey results are reported for those who completed all or most of the survey each 
time it was administered to their class. For Cohort | (fifth graders in 2015-16), 255 students 
completed all or most of the survey the three times it was administered over two school 
years. For Cohort 2 (fourth graders in 2015-16), 345 students completed all or most of the 
survey in both fall 2015 and spring 2016. Sixteen close-ended survey items were included 
at all the survey administration points. 


Small-group interviews with students 
(Questions I and 6) 


In spring 2016, small-group semi-structured interviews were conducted with fifth-grade 
students who had been participating in the program for two years. Eleven interviews 
with groups of four students (a total of 44 students) were interviewed to get in-depth 
understanding of their experiences in STEM Pathways. Two interviews were conducted 
at each of five STEM Pathways schools, and one interview was conducted at the remaining 
school. Parents were required to give consent in order for students to participate in the 
interviews. Demographic information was compiled to help ensure that the students 
selected for interviews were representative of the student population as a whole. Based 
on the desire for demographic representation, preferred interviewees were selected in 
advance of the interviews; if those students were absent from school, back-up interviewees 
were included instead. 


Academic achievement and attendance data 
(Questions 6 and 7) 


This evaluation provides student achievement results on indicators reflective of long-term 
project goals. Data are presented in the areas of math, science, and also reading, in recognition 
of the potential broader impact of STEM on student development. It is important to recognize 
that moving the needle on these indicators takes time, and numerous factors contribute to 
that effort. However, monitoring these indicators during and beyond the pilot project period 
is important in that ultimately, longer-term participation could increase MPS students’ 
success in STEM areas and narrow achievement gaps. Data are provided from the Minnesota 
Comprehensive Assessments, Series II] (MCA-IID), in math, science, and reading. 


School attendance is included in the second year evaluation as a student outcome measure. 
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Data analysis 


Minneapolis Public Schools leader interviews and STEM Pathways partner 
interviews 


Evaluators reviewed the interview transcripts and organized the respondent comments by 
key themes. As appropriate, these comments were also organized by the position or role 
of the respondents in their organization or in the project. 


Student survey 


Data are analyzed to examine changes in students’ responses over time. Response options 
to the survey items were: “don’t agree’, “agree a little”, “mostly agree”, “agree a lot’, 
and “don’t know.” Those who agreed a lot or agreed mostly were considered to be in 
agreement with an item. Statistical tests (McNemar Test, two-sided) were conducted to 
determine whether change over time in the percentage of students responding “agree a lot” 
or “mostly agree” to each item was statistically significant (p<.05). When the terms 
“significant” and “not significant” are used in describing such changes in responses, these 
terms are referring to the results of the statistical tests. For Cohort 1, the significance of 
changes in responses was assessed over the first school year (fall 2014 to spring 2015), the 
second year (spring 2015 to spring 2016), and the full two school years (fall 2014 to spring 
2016). For Cohort 2 fourth graders, changes were assessed over the 2015-16 school year, 
from fall 2015 to spring 2016. The changes occurring in Cohort 2 fourth graders’ responses 
in 2015-16 were compared to those occurring for Cohort 1 students during their fourth-grade 
year, 2014-15. 


Differences in survey responses to each survey item were also examined by student 
demographic characteristics, including gender (male, female), free- or reduced-price 
lunch eligibility (eligible, ineligible), race/ethnicity (students of color, white), and ELL 
status (ELL, non-ELL). We tested for differences in the percentage of students who 
responded “agree a lot” or “mostly agree” to each survey item within the categories of 
each characteristic (female vs. male, eligible vs. ineligible for free/reduced-price lunch, 
etc.). In both Cohort 1 and Cohort 2, we tested for these differences in spring 2016 (using 
Fisher’s Exact Test, two-sided). 


In addition, evaluators tested to determine changes in responses to each survey item over 
time among the student demographic groups. That is, we examined changes among 
males, females, those eligible for free/reduced-price lunch, those ineligible for free/ 
reduced-price lunch, students of color, white students, ELL students, and non-ELL 
students. For Cohort 1 these changes were examined for statistical significance (using 
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the McNemar Test, two-sided) from fall 2014 to spring 2016, and for Cohort 2, from fall 
2015 to spring 2016. 


Only statistically significant differences (p<.05) are reported from the analyses of change 
by student characteristics. 


Small-group interviews with students 


The evaluator reviewed interview transcripts and organized the respondent comments by 
key themes. 


Student academic achievement 


To assess student achievement, the evaluation uses a quasi-experimental design and 
analyzes data for cohorts of fourth- and fifth-grade students in the six STEM Pathways 
schools (treatment) and similar students in non-STEM Pathways schools (comparison). 
All schools are in MPS. 


The evaluation includes cohorts of fourth- and fifth-grade students in these schools in 
2014-15. Another cohort of fourth-grade students was added in 2015-16. Figure 10 shows 
the study cohorts. 


10. STEM Pathways cohort groups 


2014-15 2015-16 
Cohort 0 5'4 grade 6'" grade 
Cohort 1 4th grade 5th grade 
Cohort 2 - 4'" grade 


Academic performance of the STEM Pathways cohort students (treatment) were compared 
with those of same-grade students from non-STEM Pathways schools (comparison). The 
evaluation uses a statistical technique (propensity score matching — using exact or one-to- 
one matching) to match the treatment and comparison group students at baseline, so that 
the potential effects of STEM Pathways treatment on student outcomes are not confounded 
by differences in student characteristics between the groups that could influence student 
achievement. These characteristics include students’ prior academic achievement (MCA- 
III math achievement level in the spring prior to when treatment students received STEM 
Pathways) and several student demographic characteristics (free or reduced-price lunch 
eligibility, ELL status, special education status, gender, and race/ethnicity). 


Potential STEM Pathways effects were examined through analysis of differences between 
the treatment and comparison groups in each of the three cohorts on student achievement 
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measures: MCA-III math, reading, and science (Cohort | only) scores. Differences in 
student achievement between the treatment and comparison groups were examined in 
each cohort overall and within demographic subgroups. A detailed description of the 
research design and statistical techniques used in analyzing the student academic data and 
the findings is presented in another report, STEM Pathways Student Academic Achievement 
Results for the 2015-16 School Year (Mueller & Gozali-Lee, 2016). 


Overall strengths and limitations of methodology 


Strengths 


The study data were collected from multiple perspectives, including STEM Pathways partners, 
MPS leaders, teachers, and students. Survey and interview instruments were developed 
thoughtfully and collaboratively with the STEM Pathways steering committee and allowed 
for triangulation of respondent perspectives on several topics (e.g., partners and MPS 
leaders identified the status of collaboration). Overall, response rates were high, especially 
on the student survey. Additionally, the MPS Research, Evaluation, and Assessment (REA) 
department provided student demographics and academic achievement data for treatment 
and comparison students. 


Limitations 


Results of student surveys (i.e., changes in their responses from fall to spring related to 
STEM learning, interests, and activities) may be associated with participation in STEM 
Pathways. However, caution is needed in attributing these results to STEM Pathways as 
students may have had other STEM experiences in and out of school during the same period 
that could have also contributed to the changes. Additionally, comparisons to perspectives of 
non-participating students were not available as it was not feasible to conduct the student 
survey in a comparison group. 


Small-group interviews were conducted in English, and 25 percent of students who 
participated were English Language Learners. Although students were told the researcher 
was fluent in Spanish, they chose to speak in Spanish on only a few occasions. While the 
students seemed able to express their thoughts in English, it is possible that their responses 
may have been different if an interpreter were present. 


MCA-III tests were used to assess student academic outcomes in math, science, and 
reading proficiency. These tests are the academic achievement measures administered to 
students statewide each spring, and serve as accountability measurement tools in Minnesota. 
We consider MCA-III test results to be long-term indicators of potential effects of the 
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program. Measurement for short-term outcomes to assess student comprehension on 
specific subtopics/units of STEM was not available. 


Data privacy and research consent 


Wilder Research worked with the MPS REA to ensure evaluation procedures were consistent 
with district data privacy and research consent requirements. Wilder Research and MPS 
developed a data confidentiality agreement at evaluation onset safeguarding data security, 
and REA approved evaluation plans. REA also reviewed and approved all data collection 
instruments and consent procedures used in the course of this evaluation. 
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Implementation 


STEM Pathways’ goals for implementation include: 
= Providing students with access to multiple STEM Pathways partner programs. 


= Providing connections between STEM Pathways partner experiences via shared 
messages, vocabulary, concepts and instructional tools. 


= Offering high quality, relevant STEM experiences for STEM Pathways students. 


In both years of pilot project implementation, fourth- and fifth-grade students received 
almost 30 programming hours from STEM Pathways partners. The following are the 
grade-level program descriptions. 


Programs for students 


Fourth-grade programs 


In the fourth-grade STEM Pathways program sequence, students participate in real world 
STEM learning that helps them see a bigger picture of how STEM works and how it 
relates to them, specifically around the practices and process of science and engineering 
and STEM careers. The following organizations offer fourth-grade programs: 


The Works Museum 
Program hours: 3-hour field trip 


Students travel to the museum to explore interactive museum exhibits and participate in 
the Maze Engineering Workshop. Students use the Engineering Design Process to design 
and construct their own maze, pinball or pachinko game to take home. They experiment 
with changes in speed and direction and the effects of gravity and friction. 


The Bakken Museum 
Program hours: 1 assembly hour, 2 classroom hours and a 3.25-hour field trip 


On the first visit to the school, Bakken educators invite all fourth graders to explore what 
it means to “Wonder, Try, Discover, Share’”—important habits of every scientist and engineer. 
On the second visit, fourth-grade students apply “Wonder, Try, Discover, Share” as they re- 
invent the battery, practice supporting their claims with evidence, and learn the importance 
of taking risks and making mistakes in the process of discovery. Next, students visit The 
Bakken Museum to participate in a hands-on workshop and a guided exhibit tour to explore 
magnetism and electricity. During the tour, students also explore the history of science and 
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invention via storytelling and interactive exhibits. Every student takes home materials to 
construct an electromagnet for their own experiments. The final classroom experience is led 
by a Bakken educator and a volunteer from the STEM workforce who help students connect 
the STEM they do in school to real careers while they engage in problem solving to explore a 
real world challenge involving electromagnetism and energy. 


STARBASE Minnesota 
Program hours: 20 hours 


At STARBASE Minnesota, students learn to be engineers as they design solutions to 
challenges related to traveling from Earth to Space. As students design rockets, rover 
programs, landers, and other prototypes, including 3D printed rocket fins, they develop 
and utilize science knowledge and skills related to forming questions, conducting 
experiments, inertia, heat transfer, properties of air, and more. Students apply math skills 
such as data collection, median, measurement, estimation, and graphing as tools in the 
engineering process. 


Fifth-grade programs 


The fifth-grade STEM Pathways program sequence builds on the fourth-grade sequence 
and introduces additional connecting themes. STEM Pathways fifth-grade programs 
address the complexity and impact of change on systems. The following organizations 
offer fifth-grade programs: 


Minnesota Zoo 
Program hours: 4-hour field trip 


Through the Minnesota Zoo’s Zoo Safari program, students learn about animals and their 
habitats and participate in the Bare Necessities class. In this class, students learn about 
how ecosystems function and make predictions of impacts on habitats as a result of changes 
in ecosystems. Students gain deeper understanding of ecosystem dynamics and greater 
appreciation for the important roles everyone can play in the system. Following the 
workshop, students tour the Zoo’s exhibits. 


Bell Museum of Natural History 
Program hours: 1 classroom hour and a 4-hour field trip 


The program Honey Bees, Pollinators, and Food examines human interdependence with 
nature. Student scientists exercise critical thinking and literacy skills as they explore the 
concept of biological diversity, the process of science, and the connection between our food 
and a healthy environment. Students use scientific tools and processes—such as microscopes, 
forceps, and dissections—to make observations of the anatomy of bees and flowers. Students 
learn about current research at the University of Minnesota and apply their knowledge about 
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pollinators as they plan meals, and discuss and analyze the impact of the disappearance of 
pollinators on the meals they’ ve prepared. A tour of the Bell Museum’s dioramas allows the 
students to see some of the native habitats that are beneficial to pollinators. This program is 
structured to empower student scientists to learn more about where their food comes from, 
and how humans are connected to the many processes and relationships in obtaining food and 
maintaining a healthy environment. 


STARBASE Minnesota 
Program hours: 20 hours 


At STARBASE Minnesota, student engineers develop and utilize their knowledge of 
energy transfer and energy systems, properties of air, Newton’s Laws of Motion and 
more as they design a mission from launch, to landing, to living on Mars, including 
designing and testing 3D printed prototypes. Students apply math skills related to 
coordinate graphing, calculating mean, volume, area, and graphical data analysis as they 
complete their mission. 


MPS program 


In addition to the above programs, students can choose to participate in the MPS STEM 
program. 


GEMS/GISE 
Afterschool and summer program 


Girls in Engineering, Mathematics and Science (GEMS) and Guys In Science and 
Engineering (GISE) are elective afterschool and summer programs designed specifically 
for fourth- through eighth-grade students in all MPS schools. Project- and problem-based 
learning experiences promote integration of engineering and technology concepts, content, 
and process. Students participate in an academic culminating event involving a design 
challenge. 


Instructional tools for teachers and partners 


In addition to connecting themes and concepts, STEM Pathways programs use shared 
instructional tools and messaging (Figure 11). Vocabulary and conflict cards are intended 
for use by partners as part of STEM Pathways programs, while the career interactive, 
Student Portfolio and game of STEM are provided to teachers for use in their classrooms, 
to extend STEM learning beyond the classroom, and to promote further pursuit of STEM. 
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11. STEM Pathways Instructional Tools 


@ natural system |(@)designed system Vocabulary Cards - Common vocabulary is utilized by partner organizations, including vocabulary cards 


Conflict Cards - Used in the fifth-grade programs, each conflict card introduces a 
change to a system that students have been studying. Students are then challenged to 
determine what would happen when that change occurs. 


Student Portfolio - All students receive a STEM Pathways portfolio 
to track their STEM Pathways, classroom, and out-of-school STEM 
experiences throughout the school year. Reflection questions and 
pages to track interesting STEM careers are also included. 


Game of STEM - A game board included in student 
portfolios and a classroom poster used by teachers to recognize 


each student's STEM learning. Students “level up” as they do 
more and more STEM. 


Source: STEM Pathways 


A teacher guide was developed between the first and second implementation years, based 
on teacher feedback during the informal focus group held in the spring of 2015. This 
guide included detailed information about partner programs and instructional tools. 


In the survey, most teachers (84%) reported that they used the reflection pages of the 
STEM Pathways materials as a resource, and over 60 percent said they used career 
resources and/or activities in the Student Portfolio (Figure 12). Less than half of teachers 
(47%) said that they used the game of STEM as a resource. 
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12. Resources used by teachers 


Did you use... N Yes No 
Reflection pages 19 84% 16% 
Career resources 19 68% 32% 
Activities in the Student Portfolio 18 61% 39% 
Game of STEM 19 47% 53% 


Teachers described how they used the different resources in their classes; their responses 
are summarized below. A few teachers also mentioned some challenges in using them. 


Reflection pages: All teachers who responded ‘yes’ to this question said they used the 
reflection pages after participating in STEM Pathways programming. Teachers did not 
mention any issues with the reflection pages or give suggestions for improvement. 


Career resources: Teachers felt that the career resources were a good way for students to 
learn more about different STEM careers and they reported using career resources to 
make connections between STEM Pathways experiences and STEM professionals who 
work in related fields. Three teachers noted that they experienced technical difficulties 
while trying to view the videos. 


Student Portfolio: Teachers described the Student Portfolio as a supplemental component 
for students to work on independently, and provided specific examples of what the 
Student Portfolios covered, including circuitry and magnetism. Two teachers said that 
they did not consistently use the portfolios with their students. 


Game of STEM: Teachers thought that game of STEM had great ideas for classroom 
activities and felt that it was useful in engaging students before and after STEM Pathways 
experiences. A couple of teachers noted that it was a bit difficult and time intensive to 
use, which they said discouraged them from fully utilizing it. 
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STEM Pathways partners also reported using several tools and resources that were 
developed and prioritized by STEM Pathways to varying degrees, including: 

m= Shared vocabulary represented on vocabulary cards 

= Common concepts and themes 

= Engineering Design Processes 

= Conflict cards 


= Graphics to help students recognize and develop confidence in teamwork, creativity, 
perseverance and increased STEM skills 


m Career resources 

= Handouts with the STEM Pathways logo 

= Student Portfolio 

= Program passport with participation stamps 


The tools and resources most commonly mentioned by partner organizations as the most 
valuable to them were: 


Career resources 


| think the resources related to STEM careers have been the most valuable — easiest 
scheme to carry across from all of the organizations — easy to relate jobs to other 
organizations and the skills are broadly applicable. Those resources for teachers are also 
most valuable. — Partner interviews 


Graphics/handouts 


| think the four graphics [used at STARBASE] have been really helpful, because some of 
the stuff we work through with students is really challenging, and that can be frustrating. 
But we can remind them what they've learned and how much perseverance they have 
shown. We remind them it is part of the process and they are doing great at it, even 
though they're frustrated. - Partner interviews 


[In the] handout they receive, they see ... the STEM Pathways graphic, these big-picture 
thinking ideas that | described, using creativity and collaboration and teamwork. | think 
that’s been really useful to tie it all together. — Partner interviews 


Student Portfolio/worksheets 


For the [Student] Portfolio, we developed a take-home activity [about electromagnets]. 
The Portfolio was valuable as a reflection piece. It allowed us to get insight into how 
[students] saw their experience. — Partner interviews 
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| think the teachers really liked the worksheets that helped students reflect. They were 
part of the Portfolio. - Partner interviews 


Perspectives from partners, MPS leaders, teachers, and students 


The following section provides results from interviews with STEM Pathways partners 
and MPS leaders, the teacher survey, and small-group interviews with students in spring 
2016. The findings are organized by research questions. 


Research question 1. 
How successtully is the STEM Pathways model being implemented? 


Overall, STEM Pathways partners, MPS leaders, and teachers felt that they successfully 
implemented STEM Pathways in 2015-16. Partners felt that their commitment to working 
toward a common goal of improving student learning was a major accomplishment of this 
collaborative. MPS leaders and teachers appreciated the materials and communication 
between partners and teachers throughout the year to highlight areas of connection between 
STEM Pathways activities and the Minnesota state standards and MPS curriculum. They 
also described how STEM Pathways promoted student STEM learning and interest by 
showing them the connections between what they learn in class and STEM applications 
in the real world. In the small-group interviews, students described their STEM Pathways 
experiences as special, fun and helpful to their learning. 


Finding: Partners share a common goal of making a positive impact on students and 
value the commitment of working together. 


[The biggest accomplishment has been the] collaboration on a scale and ground level to 
move this project from concept to implementation, making a commitment to do what is 
best for kids with little incentive/funding. | think that is the greatest accomplishment in my 
mind. — Partner interviews 


It is a valuable model of how organizations can work together and accomplish more as a 
whole than as individual organizations. — Partner interviews 


| think it is the shared goal of having an impact on students. The biggest takeaway is that 
we really want to have a positive impact on these students and, by working together, we 
can. — Partner interviews 


| really think it is that commitment by each institution to ... STEM ... in this community to 
help students see themselves [as scientists and engineers] and create that STEM identity 
for their ... future[s], for filling jobs and being informed and educated citizens going 
forward. — Partner interviews 


Finding: MPS leaders felt that STEM Pathways promotes student STEM learning and 
interest by showing them the connections between what is learned in class and STEM 
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applications in the real world and, in particular, by raising their awareness of STEM- 
related careers. 


Looking through 4th grader responses, you can see several students talking about 
wanting to be an astronaut, or a mathematician, and not just typical careers like doctor or 
teacher. There are a whole lot of future prospects and career paths that have opened up. 
- MPS leader interviews 


They [STEM Pathways educators] have been great with providing resources and real-life 
applications in STEM careers and how STEM activities can help their [students’] growth 
and development as future career folks. Its been a really big positive. - MPS leader interviews 


| know that they're putting it more at the forefront. Kids are more interested in being a 
scientist or an engineer, or looking at math through a different lens, where you can apply it 
to different careers out there. - MPS leader interviews 


One of the ways the community partners amped up is what students don't get to see in 
the classroom, [which] is different careers in STEM fields. So when they go to 
organizations, they get to see scientists and engineers and mathematicians working in 
fields they may not know are available, and they get to explore the careers — have a real 
world application where they get to see engineers in action, see the careers they could be 
when they grow up. - MPS leader interviews 


Finding: STEM Pathways lessons align with Minnesota academic standards, according 
to MPS leaders, partners, and teachers. Leaders noted that partner organizations provided 
materials and communication to teachers to highlight how STEM Pathways activities 
connect with state standards and district curriculum. 


They work with the district folks to make sure what the standards are that we are going to 
focus on. People get that information and will talk with teachers, help teachers highlight 
curricular areas. It’s a team effort. Both math and science have a million standards, so | 
think these are opportunities for the partners to pull out what standards they're going to 
focus on so the teachers can do that too. - MPS leaders 


| think what | see is the alignment between standards and what’s happening in school, 
tying it to learning outside of school walls. Kids aren't learning a separate thing outside of 
school; they can see the connections between what they’ve done and work in the building 
here. We worked hard to bring alignment, and the communication [between us and 
partners] was really helpful. - MPS leader interviews 


A number of teachers have commented to me that they appreciate the way it has helped 
prepare their students better for the MCAs. | think it has been a nice way for the teachers 
to have some exciting examples for the students of the things they will get to do this year 
to bring science and engineering to life. That can be intimidating to teach, so our work can 
supplement their own lessons and strengthen their own curriculum. — Partner interviews 


Teachers consistently tell us... that the [STEM Pathways] program and curriculum are 
relevant and reach [the] standards... -— Partner interviews 


| would say by providing materials, training, coming to talk to staff at the beginning of the 
year. It's been beneficial to have reconnection through the year for questions and follow 
up. — MPS leader interviews 
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All teachers in the survey agreed or strongly agreed that STEM Pathways supports the 
state standards and the MPS curriculum and does so in meaningful ways (Figure A1). 


Finding: Students indicated that STEM Pathways experiences helped them learn and are 
fun, different from, and better than other field trips and classroom visits. 


They're way better... because it’s a fun way for me to learn. 
— Small-group interviews with students 


The STEM Pathways field trips are the ones that actually really, really helped. The other 
ones let us learn too, but we didn’t really get to do much. 
— Small-group interviews with students 


It helps you with your education, and plus you get to do things that you don’t get to do 
anywhere else. — Small-group interviews with students 


| like STEM because it’s special. You don’t get to do that every day. 
— Small-group interviews with students 


Research question 2. 


What partner and school characteristics are associated with strong implementation? In what ways 
can implementation be strengthened? 


A number of characteristics relevant to strong implementation emerged during the three- 
year pilot program. Both partners and schools indicated the importance of clear 
communication. Partners also pointed to the importance of networking, understanding 
each other’s programs, collaborative professional learning opportunities, and shared 
evaluation to implementing connected STEM learning experiences for schools. When it 
comes to working with schools, partners noted that program implementation was most 
effective at schools where STEM learning is a priority and where there is strong teacher 
and/or principal advocacy for participation in STEM Pathways. MPS teachers, principals, 
and district leaders highlighted qualities of STEM Pathways that make it a strong and 
unique program that enhances and inspires STEM learning in schools that they would like 
to see continue and expand to more students and schools. These characteristics include: 
connections to real-world applications of STEM, exposure to STEM careers and the 
broad value of STEM skills in the workplace, alignment with district curriculum and state 
standards, and inspiring excitement for STEM learning for students and teachers alike. 


At the end of the first implementation year, Wilder conducted interviews with STEM 
Pathways partners and MPS leaders to provide feedback that could help partners improve 
collaboration and implementation in the second implementation year. Partners and 
leaders reflected that more time for planning, communication, and reinforcing shared 
goals is important for successful implementation. Partners specifically mentioned interest 
in better implementing shared vocabulary and crosscutting concepts, practices, and core 
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ideas strategically across partner programs. The challenge of collaborative development 
of instructional tools was also noted—partners indicated interest in more collaborative 
involvement in the development and use of instructional tools. However, from a practical 
perspective, small development teams were required to develop tools on schedule. Partner 
educators expressed eagerness for more time to work together. MPS leaders recommended 
more communication with teachers about schedules and activities and asking for their 
feedback; teachers also made recommendations for improved communication during an 
informal focus group at the end of the first implementation year. 


These findings led partners to develop a teacher guide that included detailed logistical 
and programmatic information, improvements to the Student Portfolio, increased use of 
shared vocabulary and other instructional tools by partner educators, and a focus on 
instructional strategies to increase student confidence in their STEM abilities. 


Both partners and MPS leaders pointed to the importance of sufficient and sustainable 
funding for effective program implementation over time. 


Finding: Overall, there seems to be clearer understanding of the focus of the work during 
the second implementation year. 


Findings from the second implementation year show that MPS leaders, teachers, and 
STEM Pathways partners have clear and consistent articulations of the project’s primary 
goal, which is to increase student engagement and interest in long-term STEM learning 
and increase student interest in STEM careers. 


We offer either field trips onsite or offsite or both, opportunities at schools with science 
and engineering, and more insight into careers and what people actually do, with the hope 
of inspiring students toward careers in STEM. — Partner interviews 


| know that they're putting it more at the forefront. Kids are more interested in being a 
scientist or an engineer, or looking at math through a different lens, where you can apply it 
to different careers out there. - MPS leader interviews 


Being part of STEM Pathways has helped me to think more about the connections 
between the content | am teaching. It has also helped me link the learning in the 
classroom with the work many professionals do for their career. — Teacher survey 


Finding: MPS leaders and teachers recognize that the importance of partners’ efforts to 
foster student learning by showing them the connections between what is learned in class 
and STEM applications in the real world. 


The use of decimals at STARBASE allowed students to better see the importance of 
measuring into the hundredths and tenths; it improved their interest and understanding 
when we studied decimals in math. - Teacher survey 
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| think it goes back to real life engagement and activities... so many things to learn, how 
science fits in with the real world. Students are making those connections to core 
academics and fascinating things you do with little bits of knowledge, accessing new 
knowledge. — MPS leader interviews 


In the small-group interviews, students talked about using science, technology and 
engineering in STEM Pathways programs and named topics, such as pollination, 
electricity, and animals as specific examples of topics covered in STEM Pathways and at 
school. Students also discussed using specific math concepts (e.g., measurement, area, 
mean, and volume) both in STEM Pathways programs and at school. 


We were learning about surface area and finding the area of a 3D shape, and that really 
helped here at school. — Small-group interviews with students 


Comments from MPS leaders, teachers, and partners on ways to improve STEM 
Pathways programming can be found in the Opportunities for growth section at the end 
of the report. 


Research question 3. 


How well does the collaboration function, and how can it be strengthened? 


Overall, MPS leaders, teachers, and partners viewed the collaboration very positively. 
Examples of factors that contributed to success include: support and communication with 
teachers by STEM Pathways representatives throughout the year; soliciting of teachers 
for feedback and ideas for improvement; support of MPS curriculum; and sense of value 
of collaboration, commitment to work together, and shared goals by STEM Pathways 
partners. All responding MPS leaders and teachers expressed their wish to continue 
collaborating with the STEM Pathways partners and for STEM Pathways programming 
to continue in their schools. A few suggestions for program improvements from MPS 
leaders, teachers, and partners can be found in the Opportunities for growth section. 


Research question 4. 


How effective is professional development, and what are its future needs? 
STEM Pathways partners reported that the opportunity to network, learn about other 


informal education organizations and their programs, and learn about best practices in 
STEM education was beneficial, especially for educators at partner organizations. 
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Finding: Partners appreciate the networking opportunity with other informal education 
organizations, discussing common goals, sharing successes and challenges, and 
brainstorming new ideas. 


| really enjoy and value being able to be part of a network of my peers. Being able to, 
even if we're not always talking about STEM Pathways, but being able to network with 
other directors of education of informal science institutions in the cities... [it] has been 
really valuable to discuss challenges and discuss our successes, and look for 
commonalities in our goals and ways to strengthen each other. — Partner interviews 


Informal organizations tend to operate in isolation, even in competition with each other. 
Through STEM Pathways we were able to develop a common thread between 
organizations that don’t seem to have common relationships on the surface. 

— Partner interviews 


For me and my staff, working with other people doing similar work is very exciting and 
energizing. — Partner interviews 


Finding: Partners have gained knowledge through learning about how other informal 
education organizations operate and what their programming offers students. 


It has been an amazing awareness building experience as far as what other informal 
education organizations are doing. | learned a lot about different structures of different 
kinds of field trips. Just the difference in the way we offer our programming and design our 
field trips, understanding funding and how it works, understanding other barriers for 
organizations and schools, starting to see how a partnership like this looks in the schools 
and how it works for teachers. — Partner interviews 


It has given us opportunities to hear new ideas. Colleagues are always bouncing ideas off 
each other. It is great to get fresh perspectives on ways to approach instruction in science 
and engineering. We get a fresh look at things when we talk to partners about things we 
cover that set off little light bulbs about what we might offer; new approaches to doing 
things that we would not have thought of before. — Partner interviews 


| think the opportunity to work with the variety of different organizations. | think there was 
always this... in the past, with museums working together, there was this skepticism and 
competitive nature, | think that thinking has changed and evolved, and there is room to 
work with other institutions to get new ideas and find different ways to collaborate now. 
— Partner interviews 


All of our instructors have a greater knowledge about the other partner programs, 
concepts, and strategies, and therefore we can continue to reference those experiences 
with any student who comes through the program. — Partner interviews 


A few suggestions from MPS leaders, teachers, and partners regarding improvements in 
professional development can be found in the Opportunities for growth section. 
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Research question 5. 
What are the core components of the program model and conditions for replication? 


Interviewees from partner organizations identified the following key conditions for 
successful implementation of STEM Pathways: 


= Total buy-in and commitment from all partners, the school district, schools, and 
teachers. 


= Strong leadership and direction from the steering committee. 


= Endorsement and advocacy on the part of the district to principals and teachers at 
participating schools. 


= Small-group work (e.g., fourth- and fifth-grade cohorts, implementation team, 
steering committee). 


= Collaboration—meaningful input from every partner involved. 


= Ensuring that each partner consistently implements agreed upon programmatic 
components (e.g., in referencing the STEM Pathways Portfolio). 


= Sufficient funding. 


Through its planning process during this past year, the STEM Pathways steering 
committee has identified unique and important assets that STEM education providers 
contribute to the STEM learning ecosystem. These assets are particularly relevant for 
effective collaboration between organizations as well as partnership with schools. These 
assets include: 


= Effective instructional practices for actively engaging young people in STEM 
learning. 


= Examples of authentic real-world applications of STEM. 


= Exposure to a broad array of STEM careers and the broad utility of STEM skills in 
the workplace. 


= Development of key aspects of positive student STEM identity, such as: awareness, 
interest, engagement and confidence. 


= Meaningful connections between different STEM learning experiences offered by 
STEM education providers and in school that build on each other over time. 
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The pilot project also helped the steering committee to identify important practices for 


collaboration effectiveness and success that are central to STEM Pathways, including: 


Collaborative learning that promotes professional growth across programs and 
organizations. 


Clear and consistent messaging that communicates shared vision and goals to 
audiences and stakeholders. 


Shared instructional practices grounded in research and experience that reinforce 
desired outcomes across programs and over time. 


Information-driven decision-making that enhances STEM programs individually and 
the collaboration as a whole. 


Independent leadership that serves as a support and catalyst to enable collaborative 
process and achieve collective impact on behalf of all partners, stakeholders, and the 
community. 
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Outcomes 


The evaluation assessed short-term outcomes associated with change in student 
perceptions about STEM learning and opportunities, student school outcomes related to 
STEM Pathways long-term impacts, and overall systems change efforts. 


Student surveys and small-group interviews, a teacher survey, MPS leader interviews, 
and student academic achievement data were used to address the following questions. 


Research question 6. 


Does the STEM Pathways model enhance the short- and long-term outcomes of populations 
underrepresented in STEM? 


Research question 7. 
How well does the model work for specific underserved populations, and in what context? 


Student survey 


Evaluators used the student survey to assess changes over time in student awareness, 
interest, and confidence in STEM. A summary of the overall pattern of results across the 
two years of STEM Pathways implementation and results by topic area and student 
characteristics are presented. Figure 15 compares the results for each item in Cohort 2 to 
those in Cohort 1 in the fall and spring of fourth grade. Figure 16 shows the results for 
each item in Cohort | for those students who completed the survey all three times during 
the fourth and fifth grades. Characteristics of students participating in the survey and 
included in the analysis are presented in Figures 13 and 14. Comprehensive results can be 
found in the full report, STEM Pathways Student Survey Results for the 2015-16 School 
Year (Mueller & Gozali-Lee, 2016). 


Student demographic profiles 


Demographic profiles of Cohort 1 and 2 students in fourth grade (who completed the survey 
in both the fall and spring of fourth grade) are presented in Figure 13. These profiles are 
shown side-by-side because we will be comparing the survey results of these two cohorts 
in fourth grade. Note that the two profiles are very similar on the characteristics examined. 
In each cohort, the number of females and males is evenly divided, the distribution by 
race/ethnicity is very similar, those eligible for free/reduced-price lunch is just over three- 
quarters, those who are ELL is somewhat over one-third, and those receiving special 
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education services is about one-tenth. The lack of differences in demographic characteristics 
suggests that any differences found in survey results between the two cohorts is unlikely 
to be due to differences on these characteristics. 


13. Characteristics of fourth grade students who completed the student 
survey in both the fall and spring: 2015-16 and 2014-15 school years 


Fourth graders 


Cohort 2 Cohort 1 
2015-16 2014-15 


Student characteristics (N=345) (N=353) 
Gender Female 50% 49% 
Male 50% 51% 
Race/ethnicity American Indian 2% 2% 
African American 34% 31% 
Asian 5% T% 
Hispanic 33% 35% 
White 26% 25% 
Free/reduced-price lunch Eligible 77% 78% 
Not eligible 23% 22% 
English Language Learner (ELL) status ELL 36% 39% 
Not ELL 64% 61% 
Special education Yes 11% 9% 
No 89% 91% 


Figure 14 shows the demographic profile of Cohort 1 students who completed the student 
survey all three times (fall 2014, spring 2015, and spring 2016). Note that the number of 
these students (255) is almost 100 fewer than the number who completed the fall 2014 
and spring 2015 surveys in fourth grade (353, as shown in Figure 13). This smaller group 
of students from Cohort | are likely less mobile than the larger, original group of students 
since they completed the survey all three times, indicating that they continued to attend 
STEM Pathways schools over a two-year period. Compared to the fourth-grade group in 
Figure 13, this smaller group is somewhat less likely to be African American, eligible for 
free/reduced-price lunch, and ELL. 
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14. Characteristics of fifth graders (Cohort 1) who completed the student 
survey in fall 2014, spring 2015, and spring 2016 


Cohort 1 

Student characteristics (N=255) 
Gender Female 50% 
Male 50% 
Race/ethnicity American Indian 1% 
African American 24% 
Asian 8% 
Hispanic 38% 
White 28% 
Free/reduced-price lunch Eligible 71% 
Not eligible 29% 
English Language Learner (ELL) status ELL 33% 
Not ELL 67% 
Special education Yes 9% 
No 91% 


Overall pattern of results 


The comparison between the fourth-grade survey results of Cohort 2 in 2015-16 and 
Cohort 1 in 2014-15 indicates that overall Cohort 2 had stronger results. For example, 
Statistically significant improvement (1.e., greater agreement with survey items from fall 
to spring) occurred for 9 of the 16 survey items in Cohort 2 compared with 6 of 16 for Cohort 
1. For four of the five survey topic areas, Cohort 2 had larger increases in agreement than 
Cohort 1, higher percentages agreeing with the items at the end of the school year, or 
both. These results may be related to the STEM Pathways program being more fully 
implemented in the second year than in the first year. 


For Cohort 1, survey results were stronger overall at the end of their fifth-grade year (spring 
2016) than they were after the end of their fourth-grade year. That is, the percentages 
agreeing with the survey items at the end of fifth grade were either higher or about the 
same as they were at the end of fourth grade. Two items that did not have statistically 
significant increases in agreement at the end of fourth grade (knowing about STEM- 
related activities outside of school and liking to learn technology) did have significant 
increases by the end of fifth grade. One item that had a significant decrease in agreement 
after the first year (being really good at STEM) had a significant increase in agreement 
the second year. 
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Results by content area 


The following section summarizes the results by content areas: awareness and relevance of 
STEM, STEM interest and confidence in STEM abilities, interest in STEM subjects, 
application of STEM to problem solving, and careers using STEM. 


Awareness and relevance of STEM 


Students’ awareness of STEM and belief in its relevance increased in both cohorts. From 
fall to spring, Cohort 2 fourth graders had a significant increase in their view of the 
importance of STEM knowledge to their futures, their awareness of STEM in the world 
around them, and their knowledge of STEM-related out-of-school activities. In Cohort 1, 
agreement with the first two items also increased significantly in the fourth-grade year 
and continued to increase slightly in the fifth-grade year. Agreement with the third item 
(their knowledge of STEM-related out-of-school activities) increased significantly in fifth 
grade and was a significant increase over the two-year assessment period. Agreement 
with a fourth item in this topic area (frequently doing STEM-related out-of-school activities) 
did not change significantly in either cohort over time. 


STEM interest and confidence in STEM abilities 


Agreement with two of the four items in this topic area increased significantly from fall 
to spring among Cohort 2 fourth graders. These two items concerned liking to learn 
STEM and being really good at STEM. In Cohort 1, agreement with liking to learn 
STEM increased significantly in fourth grade but agreement with being really good at 
STEM decreased significantly. However, agreement with this latter item increased 
significantly in fifth grade so that the level of agreement at the end of fifth grade was 
nearly the same as at the beginning of fourth grade. Agreement with wanting to do more 
STEM-related activities and thinking one would be good at a job that uses STEM did not 
change significantly in either cohort over time. 


Interest in STEM subjects 


Students were asked if they liked learning each of the four STEM subjects. The percentage 
of fourth graders in Cohort 2 and Cohort | who liked learning engineering increased 
sharply from fall to spring. In Cohort 1, the increase in agreement with liking to learn 
engineering at the end of fourth grade was maintained at the end of fifth grade. Relatively 
high percentages of fourth graders in both cohorts agreed that they liked learning the 
other three STEM subjects (science, technology, and math) in both the fall and the spring. 
In Cohort 2, agreement with liking science decreased a small (but statistically significant) 
amount from fall to spring. In Cohort 1, the percentage agreeing that they liked learning 
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technology increased significantly from the beginning of fourth grade to the end of fifth 
grade. 


Application of STEM to problem solving 


Agreement of Cohort 2 fourth graders with the two items on applying STEM to problem 
solving increased significantly from fall to spring. These two items concerned using 
technology to solve problems and thinking like an engineer to design solutions to problems. 
Among fourth graders in Cohort 1, there was a significant increase in agreement with the 
engineering item but not with the technology item. The increase in the engineering item 
was maintained through fifth grade. 


Careers using STEM 


In both cohorts, fourth graders’ agreement increased significantly for the item concerning 
knowing about many jobs that use STEM. In Cohort 1, this agreement continued to increase 
in fifth grade. Agreement with two other career-related items did not change significantly 
in fourth grade in either cohort, one thinking about being good at a job that uses STEM 
(mentioned above) and another about wanting to have a job that uses STEM. Slightly 
over half of fourth graders agreed with these items. The level of agreement with these 
items did not change over the two school years in Cohort 1. 


STEM Pathways Evaluation 45 Wilder Research, December 2016 


15. STEM Pathways Student Survey: Fourth graders (Cohort 2 in 2015-16 and Cohort 1 in 


2014-15) 
Percentage of students answering “agree a lot or mostly agree” ? 
School Signif. 
Item year Number Fall Spring Test 
STEM awareness and relevance 
STEM knowledge is very important to my future. 2015-16 334 64% 16% ii 
2014-15 343 67% 15% . 
| notice STEM in the world around me every day. 2015-16 335 52% 65% ‘i 
2014-15 349 52% 61% . 
| know about many STEM-related activities that 
happen outside of school. 2015-16 334 40% 53% - 
2014-15 345 43% 45% 
| frequently do STEM-related activities outside of 
school. 2015-16 337 47% 44% 
2014-15 347 41% 40% 
STEM interest and confidence 
| like learning STEM. 2015-16 337 69% 18% i 
2014-15 346 73% 18% = 
| would like to do more STEM-related activities. 2015-16 335 62% 67% 
2014-15 348 67% 68% 
| am really good at STEM. 2015-16 338 54% 63% = 
2014-15 343 60% 48% i 
| would be good at a job that uses STEM. 2015-16 335 55% 58% 
2014-15 340 54% 54% 
Interest in STEM subjects 
| like learning math. 2015-16 336 12% 17% 
2014-15 344 72% 74% 
| like learning science. 2015-16 338 83% 18% i 
2014-15 345 81% 83% 
| like learning engineering. 2015-16 339 56% 76% . 
2014-15 339 57% 81% * 
| like learning technology. 2015-16 336 76% 80% 
2014-15 344 79% 85% 
How much do you like studying science? 
(percentage answering “quite a bit” or “very much” 2015-16 333 83% 77% * 
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15. STEM Pathways Student Survey: Fourth graders (Cohort 2 in 2015-16 and Cohort 1 in 


2014-15) (continued) 


Percentage of students answering “agree a lot or mostly agree” ? 


School Signif. 
Item year Number Fall Spring Test 
Application of STEM to problem-solving 
| use technology to solve problems. 2015-16 340 53% 64% . 
2014-15 344 55% 54% 
| think like an engineer to design solutions to problems. 2015-16 334 45% 61% if 
2014-15 339 50% 58% * 
Careers using STEM 
STEM knowledge is very important to my future. 2015-16 334 64% 76% * 
2014-15 343 67% 15% . 
| know about many jobs that use STEM. 2015-16 325 50% 68% . 
2014-15 334 57% 67% * 
| would be good at a job that uses STEM. 2015-16 335 55% 58% 
2014-15 340 54% 54% 
When | get older, | would like to have a job that uses 
STEM. 2015-16 333 50% 55% 
2014-15 343 55% 53% 


aResponse options for the survey items are: don’t agree, agree a little, mostly agree, agree a lot, and don’t know. 
Response options for the survey items are: very little, some, quite a bit, and very much, quite a bit. 


* Statistically significant (p<.05) using the McNemar Test, two-sided. 


STEM Pathways Evaluation 47 


Wilder Research, December 2016 


16. STEM Pathways Student Survey: Cohort 1, fourth and fifth grades 


Percentage of students answering 
“agree a lot or mostly agree” ? 


‘l; Signif. 2. Signif. 30 
Fall test Spring test Spring 

Item Number 2014 1 vs. 2 2015 2vs.3 2016 
STEM awareness and relevance 

STEM knowledge is very important to my future. 246 66% . 17% 81% 

| notice STEM in the world around me every day. 252 52% i 61% 65% 

| know about many STEM-related activities that happen 

outside of school. 250 40% 44% . 55% 

| frequently do STEM-related activities outside of school. 252 40% 41% 39% 
STEM interest and confidence. 

| like learning STEM. 252 15% 17% 79% 

| would like to do more STEM-related activities. 251 68% 67% 63% 

lam really good at STEM. 249 59% i 46% - 55% 

| would be good at a job that uses STEM. 246 54% 53% 53% 
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16. STEM Pathways Student Survey: Cohort 1, fourth and fifth grades (continued) 


Percentage of students answering 
“agree a lot or mostly agree” 
1. Signif. 2. Signif. 30 Signif. 
Fall test Spring test Spring test 

Item Number 2014 1 vs. 2 2015 2vs. 3 2016 1 vs. 3 
Interest in STEM subjects 

| like learning math. 249 13% 15% 15% 

| like learning science. 251 80% 82% 18% 

| like learning engineering. 244 56% * 82% 79% * 

| like learning technology. 246 19% 85% 86% 7 
Application of STEM to problem-solving 

| use technology to solve problems. 249 56% 53% 60% 

| think like an engineer to design solutions to problems. 246 50% 57% 59% * 
Careers using STEM 

STEM knowledge is very important to my future. 246 66% * 17% 81% * 

| know about many jobs that use STEM. 237 61% 69% 74% is 

| would be good at a job that uses STEM. 246 54% 53% 53% 

When | get older, | would like to have a job that uses STEM. 247 53% 53% 57% 


a Response options for the survey items are: don’t agree, agree a little, mostly agree, agree a lot, and don’t know. 
"Statistically significant (p<.05) using McNemar Test, two-sided. 
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Results by student characteristics 


Overall results of the student survey do not indicate advantages for underrepresented 
students. The survey results were examined for differences by student demographic 
characteristics (gender, free/reduced-price lunch eligibility, race/ethnicity, and ELL status). 
For many of the survey items, there were few or no differences in level of agreement by 
student characteristics in spring 2016. However, several patterns of note emerged from the 
analysis (Figures A2-A4). 


In both cohorts in spring 2016, ELL students were less likely than non-ELL students to agree 
that they: were really good at STEM, would be good at a job that uses STEM, and knew 
about many STEM jobs. In Cohort 1, students of color were less likely than white students to 
agree that they: would be good at a job that uses STEM, would like a job that uses STEM 
when they get older, and knew about many STEM jobs. These differences by ELL status and 
race/ethnicity in spring 2016 sometimes reflected baseline differences between the groups, 
and sometimes were due to differences in the change that occurred in agreement with the 
items between the groups from baseline to spring 2016. 


In Cohort 2, there were differences on a number of items between those eligible and ineligible for 
free/reduced-price lunch in spring 2016. Those eligible for free/reduced-price lunch were less 
likely than those ineligible to agree that they would be good at a job that uses STEM and that 
they knew about many STEM jobs. In addition, those eligible for free/reduced-price lunch 
were less likely than those ineligible to agree that: STEM knowledge is very important to 
their future, they notice STEM in the world around them, and they know about many out-of- 
school STEM-related activities. Finally, those eligible for free/reduced-price lunch were less 
likely than those ineligible to agree that they like learning engineering and that they think 
like an engineer to design solutions to problems. Despite these differences, Cohort 2 students 
eligible for free/reduced-price lunch increased their agreement with most of these items 
significantly from fall to spring of fourth grade. However, the increase in their agreement 
was either smaller than that of ineligible classmates, or they had lower agreement in the fall 
(baseline) than these classmates, or some of both. 


Small-group interviews with students 


Students in the small-group interviews also reported that they are more confident in their 
STEM knowledge and are more interested in STEM careers because of STEM Pathways. 
In general, STEM Pathways contributes to positive feelings about STEM. 
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Confidence in STEM 


| feel smarter because it kind of helped me know what to do and explain more. 
— Small-group interviews with students 


| feel a bit more confident in myself in science. — Small-group interviews with students 


We had tests about science that were not really going OK until we got to STEM Pathways, and 
they started helping us so we got better grades. — Small-group interviews with students 


Interest in STEM careers 


Whenever | went to any of these field trips, | always wanted to be a scientist or engineer 
person. — Small-group interviews with students 


! wanna do STEM for my job! - Small-group interviews with students 


At first | was like, ‘No, | don’t want to be a scientist in my career,’ and now I’m starting to think 
about it because after all the things we did, it’s very interesting to work with and learn about. - 
Small-group interviews with students 


Positive feelings about STEM 
No one liked technology before, but once you get in fifth grade and you do these field trips, 
you actually start to understand it. - Small-group interviews with students 


| used to think science was not fun. Now | really like science because of STEM Pathways. - 
Small-group interviews with students 


| like engineering now because you get to invent and experiment. 
— Small-group interviews with students 


MPS leader interviews and teacher survey 


MPS leaders and teachers indicated that STEM Pathways increases student enthusiasm for, 
knowledge of, and skills in STEM and provides exposure to STEM careers. MPS leaders 
recognized that STEM Pathways targets lower-income schools and focuses on closing the 
opportunity gap for these students. 


Finding: STEM Pathways has increased students’ enthusiasm for STEM, science and math 
in school. MPS leaders said that students are excited about the new knowledge and more 
engaged in school. 


Students are highly engaged with activities, with different experiences... Student love for 
science — that passionate engagement — has been high at events and during science class at 
school. — MPS leader interviews 
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| think for the students, they’re just much more enthusiastic about science and math because 
they can talk about the real life experiences that they had and it’s not just a lesson in a book or 
some problem on the page that they need to solve. Because the teachers make those explicit 
connections, | think students are a little more enthusiastic on learning science and math. - 
MPS leader interviews 


They love those field trips. When they come back, they're able to tell you what they learned, 
that they want a classroom like at STARBASE, or as an example, when they went to Bakken, 
they asked why we don’t have more of that in our science classes. It's got them more excited 
and a little more curious. — MPS leader interviews 


| think it goes back to real-life engagement and activities... so many things to learn, how 
science fits in with the real world. Students are making those connections to core academics 
and fascinating things you do with little bits of knowledge, accessing new knowledge. - MPS 
leader interviews 


Finding: STEM Pathways provides learning opportunities to students who might not have 
access to the STEM activities outside of school. 


The students in these six schools have benefitted. The opportunity to go to museums and 
educational organizations that are related to science...! don’t think every kid gets that 
experience [on their own]... | think [this] is probably the most important piece. 

— MPS leader interviews 


The trips are such a key piece, especially with so many students in poverty — they don’t have 
the opportunity to visit these places, but now they want to. - MPS leader interviews 


They have the opportunity to go to places they wouldn’t normally go. They have the 
opportunity to experience and learn some things in a more ‘real world’ way. 
— MPS leader interviews 


This program is providing opportunities that wouldn't be there for students if not for STEM 
Pathways. It’s opening a whole new world for our students; who knows who will be sparked 
and inspired to continue down this road. - MPS leader interviews 


Finding: STEM Pathways helps increase student knowledge in STEM concepts. 


Students are learning that academic language, that vocabulary, and they can then use [it] at 
school. | haven't seen as much growth in that area as I'd like, but it’s something I'd like to 
focus on. — MPS leader interviews 


Well, definitely it allowed our students to be comfortable with the knowledge and subjects that 
STEM entails. They've created strong connections between students and STEM subjects that 
they could go into deeper. It’s a highlight for all five classrooms [from our school] involved. - 
MPS leader interviews 


As mentioned earlier, MPS leaders said that STEM Pathways helps student learning by 
connecting classroom learning to the real world applications of STEM and by increasing 
student awareness of STEM-related careers. 


STEM Pathways Evaluation 52 Wilder Research, December 2016 


Similarly, all teachers in the survey were positive about the impacts of STEM Pathways on 
their students; 80 percent strongly agreed and 20 percent agreed that the program has had a 
positive impact on their students (Figure A1). In their open-ended responses, teachers most 
frequently pointed to increased student engagement and interest in learning STEM (50%) 
because of the program. Teachers also said that they have seen an improvement in student 
skill and knowledge in STEM concepts (36%), as their students have been able to directly 
apply and build upon classroom learning during the field trips. Twenty-nine percent of 
teachers spoke about their students’ increased awareness of the STEM applications in the real 
world and the resources in the community related to STEM. Teachers also mentioned increased 
exposure to new learning opportunities, engagement in cooperative group work, and 
improved confidence (14% each). 


Examples of teacher comments: 


We have seen a big increase in excitement/engagement with our core instruction, and 
students got a great deal out of the field trips that we took this year. 


My student scholars have much more confidence in the STEM areas. 


Students gain new understanding of STEM concepts and their application through the field 
trios. This exposure is not only powerful because it diversifies their learning environments and 
methods for learning, it also connects them to locations and resources within their own 
community. 


| think the Zoo hit on things we really couldn't by exposing kids to realistic habitats. 
STARBASE used technology that we don't have and was excellent at demonstrating the 
scientific method. 


Students work in cooperative groupings to improve their learning in STEM activities. The field 
trios that we did enhanced the science, math, and engineering activities we were doing on a 
daily basis. 


All of the field trips were very beneficial for student engagement and learning. 


Students are aware of STEM activities in their community. Several have said that they would 
like to return to the Works and/or The Bakken with their families. 


All teachers in the survey said that participating in STEM Pathways increased their students’ 
interest in STEM learning and STEM subjects, and 95 percent of teachers said that participating 
in STEM Pathways increased their students’ interest in STEM careers (Figure A1). 


In an open-ended question, teachers were asked how parents have reacted to their child’s 
involvement in STEM Pathways. One-third of teachers said that parents were very impressed 
by STEM Pathways and the new opportunities it gave their children. Another quarter of 
teachers said that parents mentioned how enthusiastic and engaged their children were with 
the STEM Pathways experiences. A few teachers wished for more parent involvement and 
support during the field trips. 
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Examples of teacher comments: 
Parents are generally very excited to hear their student scholar sharing their experiences while 
at home or making connections from the Pathways visits. 


Parents and families are excited and supportive of STEM Pathways. The field trips are 
highlights for them and they are excited to make the connection between STEM education and 
the current need for STEM careers. 


Parents love that students can participate in STEM exploration. They [also] do this in science 
class, and GEMS and GISE. 


Parents that have attended field trips have been very excited about our partnership. 


Parents who attended STARBASE were impressed. Perhaps we need more family 
engagement. 


Parents seem excited that their children have this opportunity. | would love to see more parent 
involvement (chaperoning, completing experiments with children) with STEM Pathways. 


Student academic achievement 


To assess the STEM Pathways potential effects on student academic achievement, treatment 
and comparison group students in each of the three cohorts are compared on the following 
measures: 


= Spring 2016 MCA-III math, science, and reading tests. Results are reported by levels 
of proficiency (does not meet standards, partially meets standards, meets standards, and 
exceeds standards) and average (mean) scale score. Students are considered proficient if 
they meet or exceed standards. Only fifth graders take the science tests. 


= One-year growth in MCA-III math, science, and reading tests. Growth, as determined 
by the Minnesota Department of Education, is reported in three categories: high, medium, 
and low growth. “High growth” is progress at a pace that will increase a student’s proficiency 
with reference to the standards over the one-year period. “Medium growth” will keep the 
student at about the same level of proficiency one-year later. “Low growth” is progress at a 
pace that will reduce the student’s proficiency over the one-year period. The one-year 
period is from spring 2015 to spring 2016. Only fifth graders take the science tests. 


= School attendance during the 2015-16 school year. Attendance is measured by the 
percentage of days attended during the whole school year. 


We consider MCA-III tests to be long-term indicators of potential effects of the program. 
Significant program effects may be unlikely in the short term, but could emerge after 
multiple years of program exposure. Data were also examined for underrepresented groups, 
which include students from low-income families, racial/ethnic minorities, and females. 
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Overall, there was no evidence found at this point for an academic achievement advantage 
among students who participated in STEM Pathways. Results indicated that differences 
between the treatment and comparison groups in spring 2016 MCA-III performance and one- 
year MCA-III growth tended to be small and not statistically significant in almost all instances. 
Although the differences were small, they generally favored the comparison group over the 
treatment group. This pattern occurred in all three cohorts for the subjects assessed (math, 
reading, and science). Only one difference was statistically significant: one-year growth in math 
was higher for comparison group fourth graders than treatment group fourth graders in 2015-16. 
Some of the differences in academic achievement in favor of the comparison groups over the 
treatment groups were stronger and statistically significant for some demographic groups. 
School attendance rates in 2015-16 were the same or almost the same in the treatment and 
comparison groups in all three cohorts. 


A full description of results and the statistical techniques used in analyzing the student 
academic data is presented in another report: STEM Pathways Student Academic Achievement 
Results Through The 2015-16 School Year (Mueller & Gozali-Lee, 2016). 


Research question 8. 
What impacts does the model have on informal STEM education organizations? 


STEM Pathways partners believe in the power of collaboration to build visible interconnections 
between the variety of STEM learning experiences that students will have over time, and that 
this will have a positive and lasting impact on student engagement and learning in STEM. 
Further, by working together, STEM Pathways partners believe that informal STEM education 
organizations can be more effective at meeting their individual goals for STEM education. 
Partners mentioned additional benefits of being part of a collaboration, including: increasing 
the visibility and credibility of participating STEM education organizations; creating common 
language and shared messages across organizations; participating in shared research and 
evaluation to support information-driven continuous improvement of programs; and enabling 
closer partnerships with schools and districts. Partners were excited about working together 
to increase attention around STEM careers and STEM skills—such as creativity, teamwork, 
and perseverance. One partner appreciated the educational materials, such as the career 
resources; another partner saw the collaborative work on writing grants and fundraising as a 
positive benefit of working together. 
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Finding: Partners believe that by having connections to other organizations, their own 
organization will increase its ability to reinforce STEM concepts and make a lasting impact 
on student STEM learning by being strongly connected to the greater whole. 


The nature of a lot of informal science education is stand-alone field trips. Students come here 
one time for hours and we can do everything in our power to make it a really powerful experience, 
but ultimately when you have this touch point, you can reach a few students in a lasting way. 
The lasting impact of the experience will be pretty minimal, but when we have something like 
STEM Pathways where they are going to continue to revisit some of the same topics in 
different ways, hopefully that makes the experiences they had here much more valuable 
because it’s part of something bigger that continues to be reinforced. - Partner interviews 


! am a strong believer that no one organization will have the silver bullet that will solve the 
STEM education challenge, so it has been satisfying because it’s envisioning the contributing 
of multiple experiences from multiple places together and how they make a difference. And to 
see how the work they do contributes to the whole. It gives a greater sense of purpose. — 
Partner interviews 


Finding: Being a network has increased the visibility and credibility of STEM organizations 
and may have the potential to attract more funders, especially given recent interest in 
collaboration in the funder community. 


It is a unique type of partnership that garners some additional visibility from the community. 
One thing that all organizations have a challenge with is growing capacity, growing visibility, 
and this type of project gives it a new spin, a new twist to how we can excite people about the 
work that we are doing in the community and in schools. — Partner interviews 


That value of collaboration and creating that ecosystem of STEM providers and then you're 
taking advantage of the credibility around that. — Partner interviews 


The collaboration was a very important attraction to funders as well. It enabled us to get 
significant funding from Boston Scientific and from Department of Defense for the study, 
project direction and other components. That was a specific result of the concept of STEM 
Pathways. PENTAIR provided funding as well. Hopefully this attracts more funding as we go 
forward. — Partner interviews 


Finding: Collaboration has led to the development of a common language around standards 
and learning targets. 


! think it’s allowed our programming to go much farther because there is this overlap with the 
5!" grade cohort, where we have this overlap of shared language and shared terminology. - 
Partner interviews 
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Finding: There is enthusiasm among partners for working together to increase attention 
around STEM careers and positive STEM attitudes and skills. 


What has been exciting is to see teammates come together and rally around STEM careers. 
That has been a big connection that we see in all of our partners and the work we have done. 
That has made our curriculum more focused on that. We always incorporated STEM careers, 
but it is more systematic, specific ways to approach it. It is exciting to see those changes in 
programming curriculum. — Partner interviews 


| think the biggest thing in the past year, the biggest value, has been in the area of how we 
address careers. We have always had careers as an underlying theme, and based on our 
involvement in STEM Pathways, we have grown to include a focused effort to address career 
skills and the soft skills of STEM - teamwork, problem solving, perseverance. — Partner 
interviews 


Finding: One partner also appreciated gaining access to educational materials, particularly 
those related to STEM careers. 


There is added value in ancillary materials, especially around career videos. — Partner 
interviews 


Finding: The partnership has given partner organizations the ability to conduct research and 
evaluation that may not be possible for individual organizations. Collaborative evaluation has 
led to access to useful, meaningful data, that allow organizations to make information-based 
program improvements. 


One of those things is related to evaluation, because if you can imagine an even larger group 
of collaborators who have agreed on important outcomes and then, after that, have found or 
developed measurements that they agree should show progress toward those outcomes, then 
that is consistent with the idea that it is not just one experience that will be the magic key, but 
rather a collection of them. Most education nonprofits do not have the resources to invest in 
evaluation on a large scale on their own, especially longitudinallly]. It increases everyone's 
capacity to access to useful, meaningful data. — Partner interviews 


{What] we learned about the evaluation of year one of implementation was eye-opening. The 
connection between the importance of students’ emotions, like confidence, enjoyment - these 
things that should be addressed more; even though students thought [organization name] was 
fun, we did not really address those aspects. As a teacher [STEM Pathways educator], it just 
changed the way | started the day. We talked more about enjoyment, rewarding aspects, 
trying to connect the fun of being at [organization name] with STEM. — Partner interviews 


The evaluation component of STEM pathways that revealed information we wouldn't have 
uncovered in any other way. We took the results of the first-year student survey to heart and 
immediately developed a whole new additional set of strategies and tools here at our 
organization to boost student confidence in STEM. — Partner interviews 
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Finding: Besides aiding research and evaluation, the partnership has led to partners working 
together to write grants and do fundraising. 


There is the research, grant writing and fundraising development work that comes with this 
initiative. That may be something that is worth calling out. - Partner interviews 


Finding: Being in the partnership allows partners to have the opportunity to work more 
closely with the MPS teachers and learn how STEM Pathways lessons fit into the curriculum 
sequence in MPS schools. 


MPS has their learning targets/objectives and it’s incredibly valuable for providers to know how 
the district thinks of standards and how they map their curriculum throughout the school year, 
and the intent of the sequence that they have created. They might have an idea about STEM 
integration and can figure out how to fit our things through the school year for students. 
Understanding their system, their language — because sometimes it’s a barrier for us. — 
Partner interviews 


Collaboration, not only with STEM Partners and colleagues at [organization name], but also 
collaborating with teachers and principals, has made the program what it is. It has been really 
fun to be in the middle of the ecosystem and learn more about all of the moving parts and how 
they fit together. — Partner interviews 


We have learned a lot about how STEM is addressed in MPS and the roles of teachers and 
specialists, which is very valuable. — Partner interviews 


Research question 9. 
What impacts does the model have on classroom teachers? 


Involvement in STEM Pathways has provided many benefits to classroom teachers, 
according to MPS leaders and teachers. STEM Pathways provides teaching materials and 
supports and inspires teachers to present the STEM curriculum to students in a way that 
motivates and excites them about STEM. 


Finding: STEM Pathways has enhanced STEM teaching by helping teachers make 
connections between STEM Pathways lessons and their classroom lessons. 


They have more authentic opportunities to engage in STEM. They can connect the math and 
science to their field trips and in moving forward with their curriculum. — MPS leader interviews 


| think it opens teachers’ eyes to opportunities that they can be making, connections that they 
might not make for their students. - MPS leader interviews 


For one, they are more mindful of looking and making those connections of “wow, this is 
something that identifies as part of STEM,” and being able to point that out to students. STEM 
is showing a high increase and need, it’s our future. - MPS leader interviews 
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Finding: STEM Pathways helps teachers integrate STEM into other areas of the curriculum. 


It helps their passion around science and the importance of science and integrated coursework. 
How do we integrate science and technology into areas like language arts and math? That's 
the really big piece I've seen. — MPS leader interviews 


We've had a strong science focus. But | think teachers have gone more in-depth in terms of 
making connections to science. For example, | was in the classroom a couple of weeks ago, 
and they made a connection with the bees and going to the Bell Museum and what U of M 
scientists are doing with the bees. | thought it was great that they were bringing that up even 
though it wasn’t the focus of the lesson. The kids were able to bring it back to that. | think that 
if anything it’s made teachers more aware of how [we are] embedding STEM into our lessons. 
— MPS leader interviews 


Finding: Students’ positive response to STEM Pathways experiences has influenced 
teachers’ attitudes about STEM teaching. 


| think the activities, lessons, journals have helped to connect with students [who were] not 
reached in more traditional approaches [and are a] benefit for teachers. ...STEM Pathways 
allows better connections to students. - MPS leader interviews 


From my perspective, seeing the kids respond so positively, it influences [the teachers’] 
attitude and their feeling about STEM. If you think about STEM from a teacher's perspective, it 
can be pretty challenging. It requires content knowledge in multiple areas. In elementary 
schools, teachers often don’t have science expertise or math expertise necessarily. They're 
more generalist educators. So when they see the students engage at that level, it causes them 
to get more excited and more interested and more connected. - MPS leader interviews 


Finding: STEM Pathways increases teacher awareness of professional development 
opportunities. 


It helps increase teacher awareness of their own lack of knowledge around STEM integration, 
or ways to integrate STEM within their classrooms — then they can reach out for more 
professional development. — MPS leader interviews 


Some teachers ... have an opportunity to extend their own learning ..., which is huge for 
teachers. One teacher ... was amazed by the resources available. - MPS leader interviews 


All teachers said that participating in STEM Pathways had a positive impact on them as a 
teacher, with 55 percent strongly agreeing that it has done so (Figure A1). In open-ended 
responses, teachers frequently responded that the STEM Pathways experience encouraged 
them to think about making real world connections with their curriculum and that it provides 
them with easy avenues to making those connections (40%). Twenty-seven percent of 
respondents spoke about how STEM Pathways has introduced them to new teaching methods 
and skills that they are able to use in the classroom. Additionally, teachers spoke about 
having access to new resources or generally enjoying teaching STEM subjects (20% each). 
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Examples of teacher comments: 


Having the opportunity to use another means of educating and connecting real-world 
knowledge with my student scholars is ALWAYS a positive addition to me, as | continue to 
develop as an educator. Each STEM site had excellent hands-on learning where | got to 
directly interact with my students, yet also allowed me the time to observe their active learning. 


! was able to learn about activities that motivate and expose my students to STEM, as well as 
have specific resources and materials to discuss future paths in STEM careers. 


Being part of STEM Pathways has helped me to think more about the connections between 
the content | am teaching. It has also helped me link the learning in the classroom with the 
work many professionals do for their career. 


It has made me conscious about helping students make real-world connections with the 
curriculum. 


Research question 10. 


What are the implications for the field of informal science education? 


MPS leaders and partners identified valuable contributions that STEM Pathways has made to 
STEM education, including the benefits of increasing student interest and knowledge in 
STEM by connecting their learning to real world applications of STEM. MPS leaders also 
felt that STEM Pathways has enhanced schools’ ability in delivering STEM education. To 
partners, STEM Pathways is a model for how informal STEM education organizations can 
work together to achieve common goals. 


Finding: STEM Pathways demonstrates the opportunity and potential for informal STEM 
programs to provide students with valuable exposure to real-life applications of STEM, 
particularly STEM-related careers. 


It helps students see what their future can be, especially for girls, and gets them to think about 
careers in math and science. | think they can see the possibility. They see themselves as 
scientists and mathematicians. - MPS leader interviews 


| think you can’t put a number or value on it; it’s so high. The more kids can see real-life 
applications to what they're learning in school - the idea of the experiential learning, learning 
about STEM, these careers that learn these skills — it’s invaluable. — MPS leader interviews 


It’s that career connection and the chance to see real-world education of the integration that 
STEM brings forward. - MPS leader interviews 


| think I’ve said it—it creates an awareness with students what STEM means, the careers ... It 
helps students understand the level of study, what needs to go into becoming a biologist or 
engineer. It reinforces the work that we’re doing here, and helps when students realize [as] 
they think back to STEM Pathways and how it applies and how they see themselves in the real 
world. They can apply what they've learned ... - MPS leader interviews 
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Finding: STEM Pathways raises awareness of the contributions informal STEM education 
programs can make in increasing student interest and engagement in STEM learning. 


STEM Pathways has helped draw attention to the important factors of student success that go 
beyond test scores. Informal organizations are good at things that build engagement and 
interest, connect to the real world. STEM Pathways has helped raise that to a higher level in 
terms of valuing that. — Partner interview 


| think that if we are looking at partnerships between formal and informal, it is a really big deal 
for schools or districts that are trying to advance STEM education curriculum... It has the 
potential to advance the understanding of what makes high-quality STEM education in both 
worlds, both bringing something unique. — Partner interview 


Finding: STEM Pathways demonstrates the potential of informal STEM education programs 
to enhance the ability of schools to provide effective STEM education for students, which is 
a benefit to students, schools, and teachers. 


STEM Pathways continues to put value on STEM topics for all students, and then it provides 
evidence — you can say you're a STEM focused school, but we can say why we're a STEM 
focused school: students have access to pathways and experiences. We have evidence to 
add to the vocabulary. - MPS leader interviews 


| think one of the challenges that both the district and the partner organizations share, along 
with almost everybody else in the United States, is that STEM is a very new paradigm. The 
idea of bringing engineering and technology, what | call the applied sciences to academic 
levels on par with science and math, is a learning curve and a retraining curve for the people, 
whether it’s a formal classroom teacher or whether it’s informal educators, to actually deliver 
STEM education that is not just STEM in name. | think the conversations between the district 
and the partner organizations have been a learning tool for both of us as to how to best do 
that. Particularly since the partner organizations are so different. And then you still need to try 
and have some of these common strategies and language. | think those have been really 
useful conversations for everyone. - MPS leader interviews 


It’s huge. It builds the awareness. It's twofold. As much as the students are benefitting, the 
teachers are totally benefitting as well. It’s not until you experience something that you think 
“how can | bring this more into my lessons, or my morning meeting, or a basic math lesson to 
bring it up to where it can embed STEM into it, not just the math parts?” That's the piece that’s 
got them excited. - MPS leader interviews 


Finding: STEM Pathways exemplifies how informal education organizations can work 
together to achieve common goals. 


It is a valuable model of how organizations can work together and accomplish more as a 
whole than as individual organizations. — Partner interview 


| think it is the shared goal of having an impact on students. The biggest takeaway is that we 
really want to have a positive impact on these students and, by working together, we can. — 
Partner interview 


STEM Pathways Evaluation 61 Wilder Research, December 2016 


Opportunities for growth 


MPS leader perspectives 


Citing the benefits and successes, all leaders said they would like STEM Pathways to 
continue, and most would like STEM Pathways to expand to other grades and schools. 


MPS leaders also discussed the importance of more coordination between teachers and 
partner organizations, as teachers could incorporate more about the field trip lessons in their 
classrooms. One leader suggested teachers and partners co-teaching during the field trips. 
Another suggested providing opportunities to differentiate instruction to accommodate 
different types of student learners. Leaders also recommended even more communication 
with teachers ahead of time regarding lessons and expectations. 


Finding: MPS leaders would like STEM Pathways to expand to other grades and schools. 


| think the benefits far outweigh the challenges. | would really like to see it expanded and | 
would really like to see it go [to] younger grades. — MPS leader interviews 


| would absolutely love to see it continue. | would also like to work beyond the initial grant that 

allows us to either expand to other community partners to offer more opportunities, or increase 
the reach to schools and students. It was written in the grant that it was pilot exploration for the 
community partners, and | think we’ve done a good job. We just want all of our students to get 
it. -— MPS leader interviews 


Yes, but | would like to see if we could expand it, particularly for our lowest socioeconomic 
schools that won't be going on any field trips unless we’re able to find them funding and 
resources. | think it’s a great opportunity for us to connect with some of our partners in the 
community. It’s also an opportunity for our students to get out of their school building and do 
learning in another setting, which tends to be very engaging. | think it’s a great program. | just 
wish it were bigger. - MPS leader interviews 


| definitely would see collaborative planning as beneficial for several sites — getting teachers 
not necessarily doing the same thing, but working toward the same goal, even if the path looks 
different. Making connections, maybe doing field trips with other [schools], so students are 
seeing other students their same age doing the same thing. - MPS leader interviews 


Finding: Leaders suggested bringing more partners into the schools or spreading out the 
partners in order to serve more schools in the district. 


Instead of having a few schools having multiple opportunities, I'd rather have all of our schools 
go on one. | know that doesn’t work very well with the way that it’s set up right now, but | would 
rather spread it out and have a group of 10 schools work with The Bakken, and a group of 10 
schools work with the Bell, and so on, so that every school would at least have an opportunity 
to do something outside of their school day that was planned and ready to go for them. - MPS 
leader interviews 
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We could spread out and not go to all community partners — have schools go to three partners, 
say, instead of all of them. Capacity-wise, | know that STARBASE can’t handle all of our 
classrooms. We can bring other community partners into the work we’ve been doing, too: 
Leonardo’s Basement, Juxtaposition, the Science Museum. Those are things we've discussed 
that could work. — MPS leader interviews 


Finding: MPS leaders recommended creating clearer expectations for teachers through 
comprehensive and collaborative planning and better programming coordination between 
teachers and partner organizations. One of the comments seems to refer to the scheduling or 
transportation issues in the first implementation year, which caused classrooms of fifth 
graders in two schools to not participate in the full STEM Pathways field trip experiences. In 
the second year, all students received all 29 hours of STEM Pathways programming. 


| always think the first challenge is the communication between partners — time to plan, time 
for teachers to get a deep understanding of what they [students] learn so they can support at 
school. — MPS leader interviews 


| think having some type of structured meeting with teachers where we have all the partners 
together so we can talk about what the year will look like, what is our goal, what's upcoming, 
so teachers can then backfill. We get to the Bell, but maybe we don’t know what we’re working 
on until a week or two before. It would be helpful to map out the year, intended learnings for 
when they [students] come out, standards they're connected to. Maybe it’s happening and I’m 
missing it. But alignment is key. It’s also important to hold schools accountable for their part, 
too — not just field trips or the extended curriculum. Being clear upfront that we need 
responses within a specific time period, when we need surveys in order to get continued 
funding, that kind of thing. Engagement with teachers would be helpful. - MPS leader 
interviews 


Sometimes the field trips make the teachers feel off the hook. Sometimes the teachers are 
supervising more and not engaged in the curriculum at the fieldtrips. | think more about, “How 
can it be a co-teaching experience when you're on the field trip?” That may be something that 
may help. — MPS leader interviews 


Finding: One leader recommended providing opportunities to differentiate instruction to 
accommodate different types of learners. 


! would think about differentiation in activities, and what students come with, to make sure 
we're meeting the needs of all students, helping all students be successful. It's hard for 
teachers, but what we know now that students are coming in with all sorts of different 
strengths and needs, differentiation is important so they can all feel as successful as their 
peers. — MPS leader interviews 


Teacher perspectives 


All teachers said that they would like their school to continue participating in the project 
(89% strongly agreed; 11% percent agreed; Figure Al). When asked to describe ways in 
which STEM Pathways could provide better support for teachers, 36 percent of respondents 
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spoke about wanting more opportunities for professional development, including on how to 
conceptualize STEM as an integrated discipline and how to incorporate the STEM Pathways 
resources into classroom learning more effectively. Eighteen percent of respondents stated 
that they wanted better access to the available resources or additional resources, including 
math and life science pages in the Student Portfolio. 


When asked to provide suggestions for how to make the STEM Pathways resources more 
valuable or engaging to students, 40 percent of respondents pointed out an area in which the 
resources were not working well. These teachers said the game of STEM was too involved 
and difficult, and one said that keeping a STEM journal makes STEM appear to be a specific 
unit of study, rather than an approach to learning. Thirty percent of respondents gave suggestions 
for additions to the materials; some ideas included providing Spanish translations, additional 
ways to recognize students for making progress, and having more STEM professionals come 
to speak to classes about their careers. 


Student interview findings 


While the overall results from the student small-group interviews are positive, the results also 
pointed to areas for program improvement. These include: 


= Students have varied understandings of STEM Pathways. Further, Students recognize 
primarily surface-level connections between STEM Pathways partners. Thus, it might be 
useful for STEM Pathways to reinforce consistent and targeted key messages for students 
and teachers about STEM Pathways and its goals, including interconnected concepts 
among partner programs. 


m= Students recognize that their STEM experiences often do not incorporate much math, and 
sometimes struggle to make connections between STEM Pathways and math. For 
programs that do not already incorporate math in a significant way, STEM Pathways may 
want to explore opportunities to incorporate math while at the same time being explicit 
when math is being used. 


m= Students recognized they still have a lot to learn about STEM. Further, the challenges 
they have learning STEM create mixed feelings related to their confidence in their 
abilities to do STEM. Thus, continued—and perhaps even increased—emphasis on 
helping students to develop attitudes and skills that build confidence may be important. 
This may include, for example, learning from mistakes, working on a team, and 
persevering through challenges. 
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Partner perspectives 


Partners had several ideas for strengthening STEM Pathways, including more opportunities 
to grow professionally. Partners also aspire to work collaboratively to develop best practices 
for STEM learning in informal settings, share outcomes and measurements, and to become 
STEM expert providers. 


Finding: Partners want more training, including training about the Next Generation Science 
Standards (NGSS); to work together with teachers to implement the training workshops; to 
continue to learn from each other by meeting longer and delving deeper on information 
sharing; and to use the evaluation results to continue developing new and improved ways of 
programming. 


We have talked a lot about mentoring staff and providing additional teacher workshop 
trainings. | think there is some potential there, but | am not sure what that is at this time. 
— Partner interviews 


One of the areas | think about is Next Generation Science Standards — as we get closer to the 
year where our own science standards are under review [and] if Minnesota adopts these [Next 
Generation Science Standards, all of the STEM Pathways organizations could benefit from 
greater training of those standards and what they look like when implemented, so when 
classrooms will implement them they might have a model to look to with our organizations. 

It would be a great area to learn about in the future so we are ready to go when they are 
implemented. — Partner interviews 


One missing piece is the classroom teachers. To me that would be a beautiful next chapter for 
this project; being able to have classroom teachers take trainings from our STEM partners and 
possibly our STEM partners continuing to get opportunities to go to workshops focused on 
STEM. Maybe more of a collaboration and plan it out together to help each other. 

— Partner interviews 


| have said a few times how great it was to have the meetings we have had. All of those 
meetings could go longer, deeper. We could do more expertise sharing, language barrier 
breaking. There is such a wealth of knowledge and passion. | wish we could do more of it and 
learn from each other. — Partner interviews 


If the project moves forward to contributing to a STEM ecosystem, there are many 
opportunities for professional growth such as tapping into the resources and taking the next 
round of data from this evaluation and collectively utilizing that information to grow in new 
ways. The needs vary from organization [to organization]. Having a bank of options for 
professional growth would be an interest to us. I'd like them based on the findings of the STEM 
Pathways [evaluation] in identifying gaps in areas of focus combined with research in the field. 
That’s the biggest priority for us. — Partner interviews 


STEM Pathways Evaluation 65 Wilder Research, December 2016 


Finding: Developing shared outcomes and metrics for STEM education is another 
opportunity for STEM Pathways partners’ growth. 


Shared measurement and metrics going forward is what | am excited about. Developing 
shared best practices of STEM education and creating professional development and 
networking opportunities around that, too. -— Partner interviews 


One of the most promising [future directions or opportunities] has to do with the idea of shared 
outcomes for student learning and growth and interest. Related to that, the way to measure it. 
How can we continue to expand on what we have done so far, working around shared 
outcomes and systems? — Partner interviews 


Finding: One partner mentioned that STEM Pathways should develop, implement, 
disseminate formal training for STEM providers and be an expert in the field. 


| feel strongly that STEM Pathways could develop and implement more formal training for 
partner staff around STEM and integrated STEM education. | am talking about developing and 
disseminating STEM education for the community and schools. We are putting ourselves out 
there as STEM expert providers. STEM Pathways has a responsibility to ensure there is 
quality. — Partner interviews 


Challenges and additional opportunity for growth 


Limited funding and partner capacity to provide programming to more schools were challenges 
most often mentioned by MPS leaders and partners. STEM Pathways partnership was still in 
its second implementation year of the pilot project, and efforts to address these challenges are 
underway. 


Limited funding 


For our site, the funding is definitely a challenge. With the number of field trips, bussing gets 
expensive, especially when you have a student demographic where 80% qualify for free and 
reduced lunch. That is an expense that we're asking of families, and we appreciate it even 

when they only give a little, but it’s an expense for us and for them. — MPS leader interviews 


Funding. We have had to tell the schools that we cannot offer it to all of these students for no 
cost anymore. | have had two schools who have had to opt out because they cannot afford it 
for the 2016-2017 [school] year, and that is even cutting the cost in half, which does not even 
cover our costs for the program. This is really expensive, so how can we work individually or 
together to secure funding. — Partner interviews 


We all, at the beginning, agreed to find ways and resources within our own organizations to 
really start this partnership. We are really at this crossroads where we would all like to 
continue, but where is the funding coming from? Where are we going to find the resources? 
We have had to make some changes this year because of that. A lot of us have shifted from 
providing a free program to providing a discounted program. — Partner interviews 
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Partner capacity to offer programming 


| think that the [capacity] size is one challenge. These partner organizations are relatively 
small. Their capacity to deliver programming even if we had the funding would have to be very 
different. - MPS leader interviews 


To be honest my concern about the program has always been that it’s in six of our schools 
and | need it in 44. So it’s not very equitable. Whoever ends up in one of these schools has 
access to this and if they don’t end up in this school, they don't. ...! realize there’s a financial 
component, but | would prefer that we were a STEM Pathways district as opposed to a set of 
STEM Pathways schools. - MPS leader interviews 


| think the other big question is the capacity. What is our capacity? And defining that if we want 
to expand beyond Minneapolis and if we want to bring on other partners as STEM Pathways 
partners. | think some deep discussions need to be had, and we need to figure out what we 
can do on an annual basis and what the ultimate goal is. — Partner interviews 


The way that this program could support is through funding. We just need resources. And to 
do that, someone to really be able to have the time and ownership to lead and manage the 
program. Right now, it’s an add-on to a lot of people’s jobs, and we’ve been able to make it 
happen, but the time allowed to do it can be challenging to find. — Partner interviews 


Partners have been proactively searching for more funding by applying for grants. To 
increase capacity to serve more students and schools, one partner suggested adding more 
partners into the collaborative, which was also suggested by MPS leaders. Another partner 
said that roles and duties in project implementation and distribution of funding should be 
clarified. 


We just applied for this Bush Foundation grant, and if we get it, it will allow us to look at the 
evaluation we have received and the bigger goals; it will give us that foundation we need to 
move forward. We have been trying to get this strategic planning with big goals, and some 
solid funding would give us the opportunity to solidify where we are and start looking for these 
bigger goals for creating an ecosystem. — Partner interviews 


If we get to the point of discussing scaling it up, offering it to the whole school district or 
multiple districts, | think that will offer a pretty huge challenge. | would like to hear from other 
projects like this about how they scale it up and the ecosystem idea where it is not necessarily 
always the same partners. It might be a bunch of different partners and schools are able to 
pick and choose or partners can come and go, but ultimately we are all supporting the same 
goal. | think that is a really interesting idea. — Partner interviews 


Just to ensure that when there is funding opportunities going forward, that there is ... equitable 
distribution of those funds amongst partner organizations, and that if there are specific goals or 
needs of different partner organizations, that those goals and needs are very clearly 
articulated. — Partner interviews 
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Additional comments 


Following are additional comments from all responding teachers. 


Excellent program with great resources. 
You give us great opportunities. Our science kits in MPS are pretty weak in fifth grade. 


!am retiring this year and | have enjoyed working at [school] with STEM Pathways. | feel strongly 
that students learn best when they are actively engaged. STEM activities are terrific to get kids 
learning in fun ways. We couldn't do it without the support from the STEM Pathways staff! 


| really believe it is a worthwhile endeavor, and it's great to feel like the content we are 
teaching is being supported and reinforced when we head out to different field trips. 


This has been a great program for [school] students and teachers. One aspect that has made 
it so amazing is that costs have been greatly subsidized. Many of the field trips and visits 
would not be possible without this funding. Any way to maintain or even increase the amount 
of funding designated to subsidizing costs for schools would help make this accessible to us 
and our students. 


Partners’ comments most often restated their appreciation for commitment of all the 
organizations involved in the project. 


| think it is a fabulous program. We just need to be able to identify ways that we can really 
support this program — support the organizations that are delivering these programs. That is 
how we will all have a greater impact on the students we serve. 


! am so thankful. | appreciate the in-kind donation of time and expertise from all of the 
partners. A lot have gone above and beyond what was being paid for. 
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Looking forward 


Having completed the three-year pilot project, partners continue to focus on planning and 
action for the next phase, building from the successes and lessons learned from the STEM 
Pathways pilot project and research from the field. There is strong consensus among partners 
regarding the value and potential impact of collaboration. 


STEM Pathways partners share a vision for providing programming that raises STEM 
awareness, interest, enthusiasm and confidence and pursuit of STEM careers. Further, partners 
recognize the importance of and are committed to fostering a collaborative systems approach 
to inspire the next generation of STEM-literate decision-makers and problem solvers. The 
goal of STEM Pathways is to ensure that all young people have opportunities to engage in 
high-quality STEM learning experiences—over time and in a variety of settings—that lead 
them to develop the STEM mindset necessary to become the creative, STEM-skilled thinkers 
and innovators of the future. Core activities of the next stage of STEM Pathways include: 


= Supporting and expanding a robust network and community of practice of informal 
STEM educators. 


= Facilitating access to a system of high-quality and interconnected STEM learning 
experiences for youth provided by informal STEM education organizations, in 
collaboration with schools and districts and alignment with and support of core 
curriculum and standards. 


= Promoting cross-organizational leadership to create and prioritize a culture of 
collaboration that builds authentic connections between organizations, people and 
programs; that articulates shared vision and goals; and that utilizes shared measurement 
to evaluate progress. 


= Building evaluation capacity of the collaboration and partner organizations, including 
development of a shared measurement system, to encourage the use of evaluation 
findings to inform decision-making and program enhancements. To this end, STEM 
Pathways is particularly interested in measures that relate to STEM interest, mindset 
(confidence, perseverance, creativity, team work), and careers. 


= Building effective funding and sustainability strategies. 
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Appendix 


A1. Teacher perspectives of STEM Pathways 


Strongly Strongly 

N Agree Agree Disagree Disagree 
Participating in STEM Pathways had a 
positive impact on my school. 20 80% 20% 0% 0% 
Participating in STEM Pathways had a 
positive impact on me as a teacher. 20 55% 45% 0% 0% 
Participating in STEM Pathways had a 
positive impact on my students. 20 80% 20% 0% 0% 
| made connections for my students 
between the different STEM Pathways 
learning experiences. 19 47% 42% 11% 0% 


| made connections for my students 
between the STEM Pathways 
experiences and classroom learning. 19 37% 63% 0% 0% 


STEM Pathways supports the 
Minneapolis Public School district's 
curriculum in meaningful ways. 19 47% 53% 0% 0% 


STEM Pathways supports Minnesota’s 
academic standards in meaningful ways. 19 58% 42% 0% 0% 


Participating in STEM Pathways 
increased my students’ interest in STEM 
learning. 19 79% 21% 0% 0% 


Participating in STEM Pathways 
increased my students’ interest in STEM 


careers. 19 63% 32% 5% 0% 
Participating in STEM Pathways improved 
my students’ learning in STEM subjects. 19 58% 42% 0% 0% 


My students made connections between 
the different STEM Pathways learning 
experiences. 19 42% 53% 5% 0% 


Overall, the STEM Pathways partner 
experiences were engaging for my 


students. 19 79% 21% 0% 0% 
| would like my school to participate in 
STEM Pathways again next year. 19 89% 11% 0% 0% 
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A2. STEM Pathways student survey fourth grade (Cohort 2) results by student characteristics, 2015-16: Statistically 
significant results (p<.05) by survey item 


Significant change within group in percentage who 


signineant difference Between Groups “agree a lot” or “mostly agree” with survey item *° 


in percentage who “agree a lot” or “mostly 


Student characteristics agree” with survey item in Spring 2016 2° Group Fall 2015 — Spring 2016 
STEM knowledge is very important to my future. 
Gender Female (n=166) 64% 74% 
Male (n=168) 63% 18% 
Free/reduced-price lunch —_ Eligible (n=255) 73% Ineligible (n=79) 85% | Eligible (n=255) 63% 73% 
Ineligible (n=79) 66% 85% 
Race/ethnicity Of color (n=247) 63% 75% 
White (N=87) 67% 19% 
ELL status ELL (n=121) 58% 73% 
Non-ELL (n=213) 67% 78% 
| notice STEM in the world around me every day. 
Gender Male (n=166) 50% 68% 
Free/reduced-price lunch _—_ Eligible (n=255) 62% Ineligible (n=80) 78% | Eligible (n=255) 49% 62% 
Ineligible (n=80) 61% 18% 
Race/ethnicity Of color (n=247) 51% 65% 
ELL status ELL (n=122) 55%  Non-ELL (n=213) 71% _ | Non-ELL (n=213) 56% 71% 
| know about many STEM.-related activities that happen outside of school. 
Gender Female (n=166) 36% 51% 
Male (n=168) 43% 54% 
Free/reduced-price lunch _—_ Eligible (n=255) 49% Ineligible (n=79) 63% | Eligible (n=255) 37% 49% 
Ineligible (n=79) 47% 63% 
Race/ethnicity Of color (n=246) 37% 51% 
ELL status Non-ELL (n=212) 42% 56% 
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A2. STEM Pathways student survey fourth grade (Cohort 2) results by student characteristics, 2015-16: Statistically 
significant results (p<.05) by survey item (continued) 


Significant difference between groups Significant change within group in percentage who 
in percentage who “agree a lot” or “mostly “agree a lot” or “mostly agree” with survey item 7° 
Student characteristics agree” with survey item in Spring 2016 2” Group Fall 2015 Spring 2016 
| like learning STEM. 
Gender Female (n=169) 70% 80% 
Free/reduced-price lunch Ineligible (n=80) 68% 84% 
Race/ethnicity Of color (n=249) 68% 78% 
ELL status ELL (n=123) 63% 78% 
lam really good at STEM. 
Free/reduced-price lunch Ineligible (n=79) 49% 12% 
Race/ethnicity White (n=88) 56% 69% 
ELL status ELL (n=122) 54% Non-ELL (n=216) 68% | Non-ELL (n=216) 56% 68% 


| would be good at a job that uses STEM. 
Free/reduced-price lunch _ Eligible (n=256) 54% Ineligible (n=79) 68% 


ELL status ELL (n=122) 50%  Non-ELL (n=213) 62% 
| like learning math. 
Gender Male (n=170) 65% 75% 
ELL status ELL (n=124) 69% 80% 
| like learning science. 
ELL status Non-ELL (n=214) 85% 78% 
| like learning engineering. 
Gender Female (n=168) 54% 73% 
Male (n=171) 58% 78% 
Free/reduced-price lunch _—_ Eligible (n=259) 72% Ineligible (n=80) 88% | Eligible (n=259) 54% 72% 
Ineligible (n=80) 65% 88% 
Race/ethnicity Of color (n=251) 53% 73% 
White (n=88) 65% 82% 
ELL status ELL (n=123) 47% 70% 
Non-ELL (n=216) 62% 79% 
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A2. STEM Pathways student survey fourth grade (Cohort 2) results by student characteristics, 2015-16: Statistically 


significant results (p<.05) by survey item (continued) 


Significant difference between groups 
in percentage who “agree a lot” or “mostly 


Significant change within group in percentage who 
“agree a lot” or “mostly agree” with survey item 7° 


Student characteristics agree” with survey item in Spring 2016 2” Group Fall 2015 Spring 2016 
| like learning technology. 
Free/reduced-price lunch Eligible (n=258) 78% Ineligible (n=78) 88% 
| use technology to solve problems. 
Gender Female (n=170) 47% 63% 
Free/reduced-price lunch Eligible (n=260) 54% 64% 
Ineligible (n=80) 49% 65% 
Race/ethnicity Of color (n=252) 52% 64% 
ELL status ELL (n=124) 51% 65% 
Non-ELL (n=216) 54% 64% 
| think like an engineer to design solutions to problems. 
Gender Female (n=168) 43% 58% 
Male (n=166) 47% 64% 
Free/reduced-price lunch _—_ Eligible (n=256) 56% Ineligible (n=78) 77% _ | Eligible (n=256) 43% 56% 
Ineligible (n=78) 51% 77% 
Race/ethnicity Of color (n=246) 46% 59% 
White (n=88) 42% 65% 
ELL status Ell (n=122) 43% 57% 
Non-ELL (n=212) 46% 63% 
| know about many jobs that use STEM. 
Gender Female (n=163) 50% 64% 
Male (n=162) 51% 13% 
Free/reduced-price lunch _—_ Eligible (n=245) 64% Ineligible (n=80) 83% | Eligible (n=245) 47% 64% 
Ineligible (n=80) 61% 83% 
Race/ethnicity Of color (n=237) 45% 66% 
ELL status ELL (n=116) 61% Non-ELL (n=209) 72% | ELL (n=116) 38% 61% 
Non-ELL (n=209) 57% 72% 
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A2. STEM Pathways student survey fourth grade (Cohort 2) results by student characteristics, 2015-16: Statistically 
significant results (p<.05) by survey item (continued) 


Significant change within group in percentage who 


Significant difference: between groups “agree a lot” or “mostly agree” with survey item *° 


in percentage who “agree a lot” or “mostly 


Student characteristics agree” with survey item in Spring 2016 2” Group Fall 2015 — Spring 2016 
How much do you like studying science? Percent responding quite a bit/very much: 4 
ELL status ELL (n=121) 84% Non-ELL (n=212) 73% Non-ELL (n=212) 81% 73% 


@Response options for the survey items are: don’t agree, agree a little, mostly agree, agree a lot, and don’t know. 
> Statistically significant difference (p<.05) using Fisher’s Exact Test, two-sided. 

¢ Statistically significant change (p<.05) using McNemar Test, two-sided. 

4Response options for the survey items are: very little, some, quite a bit, and very much, quite a bit. 


A3. STEM Pathways Student Survey: Cohort 1, fourth and fifth grades 


Percentage of students answering 
“agree a lot or mostly agree” ? 
1. Signif. 2. Signif. 30 Signif. 
Fall test Spring test Spring test 

Item Number 2014 1vs. 2 2015 2vs.3 2016 1 vs. 3 
STEM awareness and relevance 

STEM knowledge is very important to my future. 246 66% = 17% 81% i 

| notice STEM in the world around me every day. 252 52% . 61% 65% : 

| know about many STEM-related activities that happen 

outside of school. 250 40% 44% * 55% ‘i 

| frequently do STEM-related activities outside of school. 252 40% 41% 39% 
STEM interest and confidence. 

| like learning STEM. 252 15% 17% 79% 

| would like to do more STEM-related activities. 251 68% 67% 63% 

lam really good at STEM. 249 59% ‘ 46% - 55% 

| would be good at a job that uses STEM. 246 54% 53% 53% 
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A3. STEM Pathways Student Survey: Cohort 1, fourth and fifth grades (continued) 


Percentage of students answering 
“agree a lot or mostly agree” ? 


1. Signif. 2. Signif. 30 Signif. 
Fall test Spring test Spring test 

Item Number 2014 1 vs. 2 2015 2vs.3 2016 1vs. 3 
Interest in STEM subjects 

| like learning math. 249 13% 15% 15% 

| like learning science. 251 80% 82% 18% 

| like learning engineering. 244 56% * 82% 79% * 

| like learning technology. 246 19% 85% 86% * 
Application of STEM to problem-solving 

| use technology to solve problems. 249 56% 53% 60% 

| think like an engineer to design solutions to problems. 246 50% 57% 59% : 
Careers using STEM 

STEM knowledge is very important to my future. 246 66% * 17% 81% * 

| know about many jobs that use STEM. 237 61% 69% 74% m 

| would be good at a job that uses STEM. 246 54% 53% 53% 

When | get older, | would like to have a job that uses STEM. 247 53% 53% 57% 


aResponse options for the survey items are: don’t agree, agree a little, mostly agree, agree a lot, and don’t know. 


*Statistically significant (p<.05) using McNemar Test, two-sided. 
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A4. STEM Pathways student survey Cohort 1 results by student characteristics, Fall 2014 to Spring 2016: Statistically 
significant results (p<.05) by survey item 


Significant difference between groups Significant change within group in percentage who 
in percentage who “agree a lot” or “mostly “agree a lot” or “mostly agree” with survey item 7° 
Student characteristics agree” with survey item in Spring 2016 2° Group Fall 2014 Spring 2016 
STEM knowledge is very important to my future. 
Gender Female (n=124) 62% 83% 
Free/reduced-price lunch Eligible (n=177) 63% 82% 
Race/ethnicity Of color (n=179) 67% 80% 
White (N=67) 63% 84% 
ELL status ELL (n=81) 59% 77% 
Non-ELL (n=165) 69% 83% 
| notice STEM in the world around me every day. 
Gender Female (n=126) 52% 65% 
Male (n=126) 52% 65% 
Free/reduced-price lunch Eligible (n=180) 51% 64% 
Race/ethnicity Of color (n=181) 52% 67% 
ELL status ELL (n=82) p=.05 48% 61% 
Non-ELL (n=170) 54% 67% 
| know about many STEM.-related activities that happen outside of school. 
Gender Female (n=124) 34% 54% 
Free/reduced-price lunch Eligible (n=178) 40% 55% 
Race/ethnicity White (n=70) 33% 60% 
ELL status Non-ELL (n=167) 40% 59% 
lam really good at STEM. 
ELL status ELL (n=83) 42% Non-ELL (n=166) 62% 
| would be good at a job that uses STEM. 
Race/ethnicity Of color (n=175) 49% White (n=71) 63% 
ELL status ELL (n=81) 41% Non-ELL (n=165) 59% 


| like learning math. 
Free/reduced-price lunch _—_ Eligible (n=176) 80% Ineligible (n=73) 63% 
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A4. STEM Pathways student survey Cohort 1 results by student characteristics, Fall 2014 to Spring 2016: Statistically 


significant results (p<.05) by survey item (continued) 


Significant difference between groups 
in percentage who “agree a lot” or “mostly 


Significant change within group in percentage who 
“agree a lot” or “mostly agree” with survey item 7° 


Student characteristics agree” with survey item in Spring 2016 2° Group Fall 2014 Spring 2016 
| like learning engineering. 
Gender Female (n=123) 46% 75% 
Male (n=121) 66% 83% 
Free/reduced-price lunch Eligible (n=172) 55% 79% 
Ineligible (n=72) 57% 18% 
Race/ethnicity Of color (n=176) 57% 78% 
White (n=68) 53% 79% 
ELL status ELL (n=79) 48% 78% 
Non-ELL (n=165) 59% 19% 
| like learning technology. 
Free/reduced-price lunch Eligible (n=174) 79% 87% 
| know about many jobs that use STEM. 
Gender Female (n=122) 55% 73% 
Free/reduced-price lunch Eligible (n=166) 59% 12% 
Ineligible (n=71) 66% 19% 
Race/ethnicity Of color (n=167) 69% White (n=70) 86% | Of color (n=167) 57% 69% 
White (n=70) 70% 86% 
ELL status ELL (n=76) 64%  Non-ELL (n=161) 79% | Non-ELL (n=161) 64% 79% 
When | get older, | would like to have a job that uses STEM. 
Free/reduced-price lunch Ineligible (n=72) 49% 65% 
Race/ethnicity Of color (n=176) 52% White (n=71) 69% | White (n=71) 49% 69% 
ELL status Non-ELL (n=166) 50% 61% 


aResponse options for the survey items are: don’t agree, agree a little, mostly agree, agree a lot, and don’t know. 


> Statistically significant difference (p<.05) using Fisher's Exact Test, two-sided. 
¢ Statistically significant change (p<.05) using McNemar Test, two-sided. 
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